Jul 02 13:12:30.984389 ip-10-0-135-143 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 02 13:12:30.984398 ip-10-0-135-143 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 02 13:12:30.984405 ip-10-0-135-143 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 02 13:12:30.984628 ip-10-0-135-143 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 02 13:12:41.233605 ip-10-0-135-143 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 02 13:12:41.233625 ip-10-0-135-143 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot a2eae737eef046738d8f76ade3e79b59 --
Jul 02 13:14:47.479273 ip-10-0-135-143 systemd[1]: Starting Kubernetes Kubelet...
Jul 02 13:14:49.073093 ip-10-0-135-143 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 13:14:49.073093 ip-10-0-135-143 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 02 13:14:49.073093 ip-10-0-135-143 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 13:14:49.073093 ip-10-0-135-143 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 02 13:14:49.073093 ip-10-0-135-143 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 13:14:49.074893 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.074772    2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 02 13:14:49.080595 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080569    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:14:49.080595 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080589    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:14:49.080595 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080593    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:14:49.080595 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080597    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:14:49.080595 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080601    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:14:49.080595 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080604    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:14:49.080830 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080607    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:14:49.080830 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080610    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:14:49.080830 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080614    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:14:49.080830 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080617    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:14:49.080830 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080620    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:14:49.080830 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080622    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:14:49.080830 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080625    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:14:49.080830 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080628    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:14:49.080830 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080630    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:14:49.080830 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080633    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:14:49.080830 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080636    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:14:49.080830 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080640    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:14:49.080830 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080644    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:14:49.080830 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080647    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:14:49.080830 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080650    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:14:49.080830 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080653    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:14:49.080830 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080656    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:14:49.080830 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080658    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:14:49.080830 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080661    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:14:49.081273 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080665    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:14:49.081273 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080671    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:14:49.081273 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080674    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:14:49.081273 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080677    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:14:49.081273 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080680    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:14:49.081273 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080683    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:14:49.081273 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080686    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:14:49.081273 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080688    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:14:49.081273 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080691    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:14:49.081273 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080693    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:14:49.081273 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080696    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:14:49.081273 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080698    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:14:49.081273 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080701    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:14:49.081273 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080703    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:14:49.081273 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080707    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:14:49.081273 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080709    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:14:49.081273 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080712    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:14:49.081273 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080714    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:14:49.081273 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080717    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:14:49.081273 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080720    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:14:49.081816 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080723    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:14:49.081816 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080725    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:14:49.081816 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080728    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:14:49.081816 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080730    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:14:49.081816 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080733    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:14:49.081816 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080736    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:14:49.081816 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080740    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:14:49.081816 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080744    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:14:49.081816 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080747    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:14:49.081816 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080750    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:14:49.081816 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080753    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:14:49.081816 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080755    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:14:49.081816 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080758    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:14:49.081816 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080761    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:14:49.081816 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080763    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:14:49.081816 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080766    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:14:49.081816 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080768    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:14:49.081816 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080771    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:14:49.081816 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080773    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:14:49.081816 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080776    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:14:49.082292 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080778    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:14:49.082292 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080781    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:14:49.082292 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080785    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:14:49.082292 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080787    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:14:49.082292 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080790    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:14:49.082292 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080792    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:14:49.082292 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080796    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:14:49.082292 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080799    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:14:49.082292 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080802    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:14:49.082292 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080804    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:14:49.082292 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080807    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:14:49.082292 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080809    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:14:49.082292 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080812    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:14:49.082292 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080814    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:14:49.082292 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080817    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:14:49.082292 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080819    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:14:49.082292 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080822    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:14:49.082292 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080824    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:14:49.082292 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080827    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:14:49.082292 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080830    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:14:49.082815 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080833    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:14:49.082815 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080836    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:14:49.082815 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080839    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:14:49.082815 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080842    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:14:49.082815 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080844    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:14:49.082815 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080847    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:14:49.082815 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080850    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:14:49.082815 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080852    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:14:49.082815 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080855    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:14:49.082815 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080857    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:14:49.082815 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080860    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:14:49.082815 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080862    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:14:49.082815 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080865    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:14:49.082815 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.080867    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:14:49.082815 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081612    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:14:49.082815 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081619    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:14:49.082815 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081623    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:14:49.082815 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081625    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:14:49.082815 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081629    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:14:49.082815 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081632    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:14:49.083301 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081635    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:14:49.083301 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081638    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:14:49.083301 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081640    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:14:49.083301 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081643    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:14:49.083301 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081646    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:14:49.083301 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081649    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:14:49.083301 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081651    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:14:49.083301 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081654    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:14:49.083301 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081657    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:14:49.083301 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081660    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:14:49.083301 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081663    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:14:49.083301 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081666    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:14:49.083301 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081669    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:14:49.083301 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081671    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:14:49.083301 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081674    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:14:49.083301 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081677    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:14:49.083301 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081679    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:14:49.083301 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081682    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:14:49.083301 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081685    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:14:49.083301 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081687    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:14:49.083807 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081690    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:14:49.083807 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081692    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:14:49.083807 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081695    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:14:49.083807 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081697    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:14:49.083807 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081701    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:14:49.083807 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081705    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:14:49.083807 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081708    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:14:49.083807 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081710    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:14:49.083807 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081713    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:14:49.083807 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081715    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:14:49.083807 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081719    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:14:49.083807 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081722    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:14:49.083807 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081725    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:14:49.083807 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081727    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:14:49.083807 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081730    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:14:49.083807 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081732    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:14:49.083807 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081735    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:14:49.083807 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081737    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:14:49.083807 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081740    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:14:49.083807 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081742    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:14:49.084286 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081745    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:14:49.084286 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081747    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:14:49.084286 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081750    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:14:49.084286 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081752    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:14:49.084286 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081756    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:14:49.084286 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081759    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:14:49.084286 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081761    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:14:49.084286 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081764    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:14:49.084286 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081766    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:14:49.084286 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081769    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:14:49.084286 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081771    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:14:49.084286 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081774    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:14:49.084286 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081776    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:14:49.084286 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081780    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:14:49.084286 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081782    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:14:49.084286 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081785    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:14:49.084286 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081788    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:14:49.084286 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081791    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:14:49.084286 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081794    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:14:49.084286 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081796    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:14:49.084786 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081799    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:14:49.084786 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081802    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:14:49.084786 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081804    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:14:49.084786 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081807    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:14:49.084786 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081809    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:14:49.084786 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081812    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:14:49.084786 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081814    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:14:49.084786 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081817    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:14:49.084786 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081819    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:14:49.084786 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081822    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:14:49.084786 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081825    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:14:49.084786 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081828    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:14:49.084786 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081831    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:14:49.084786 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081835    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:14:49.084786 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081838    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:14:49.084786 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081841    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:14:49.084786 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081844    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:14:49.084786 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081847    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:14:49.084786 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081849    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:14:49.085242 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081852    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:14:49.085242 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081855    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:14:49.085242 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081861    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:14:49.085242 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081864    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:14:49.085242 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081867    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:14:49.085242 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081870    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:14:49.085242 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081873    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:14:49.085242 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081875    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:14:49.085242 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081878    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:14:49.085242 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081881    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:14:49.085242 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081884    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:14:49.085242 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081887    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:14:49.085242 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081890    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:14:49.085242 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.081892    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:14:49.085242 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082359    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:14:49.085242 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082366    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:14:49.085242 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082370    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:14:49.085242 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082374    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:14:49.085242 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082377    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:14:49.085242 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082380    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:14:49.085766 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082399    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:14:49.085766 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082404    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:14:49.085766 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082406    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:14:49.085766 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082409    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:14:49.085766 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082412    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:14:49.085766 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082415    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:14:49.085766 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082417    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:14:49.085766 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082420    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:14:49.085766 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082422    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:14:49.085766 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082425    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:14:49.085766 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082428    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:14:49.085766 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082431    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:14:49.085766 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082433    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:14:49.085766 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082436    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:14:49.085766 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082439    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:14:49.085766 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082442    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:14:49.085766 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082445    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:14:49.085766 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082448    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:14:49.085766 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082450    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:14:49.085766 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082453    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:14:49.086241 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082455    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:14:49.086241 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082458    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:14:49.086241 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082462    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:14:49.086241 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082466    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:14:49.086241 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082469    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:14:49.086241 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082472    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:14:49.086241 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082475    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:14:49.086241 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082480    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:14:49.086241 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082483    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:14:49.086241 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082486    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:14:49.086241 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082488    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:14:49.086241 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082491    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:14:49.086241 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082494    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:14:49.086241 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082497    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:14:49.086241 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082499    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:14:49.086241 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082502    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:14:49.086241 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082504    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:14:49.086241 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082506    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:14:49.086241 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082509    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:14:49.086707 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082511    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:14:49.086707 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082514    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:14:49.086707 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082516    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:14:49.086707 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082519    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:14:49.086707 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082522    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:14:49.086707 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082524    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:14:49.086707 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082527    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:14:49.086707 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082529    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:14:49.086707 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082532    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:14:49.086707 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082535    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:14:49.086707 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082539    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:14:49.086707 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082541    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:14:49.086707 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082544    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:14:49.086707 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082546    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:14:49.086707 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082552    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:14:49.086707 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082554    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:14:49.086707 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082557    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:14:49.086707 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082559    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:14:49.086707 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082561    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:14:49.086707 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082564    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:14:49.087219 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082566    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:14:49.087219 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082568    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:14:49.087219 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082571    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:14:49.087219 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082573    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:14:49.087219 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082576    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:14:49.087219 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082578    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:14:49.087219 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082581    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:14:49.087219 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082583    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:14:49.087219 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082586    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:14:49.087219 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082588    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:14:49.087219 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082591    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:14:49.087219 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082593    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:14:49.087219 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082596    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:14:49.087219 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082598    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:14:49.087219 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082601    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:14:49.087219 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082603    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:14:49.087219 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082606    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:14:49.087219 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082609    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:14:49.087219 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082611    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:14:49.087219 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082614    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:14:49.087731 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082618    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:14:49.087731 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082622    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:14:49.087731 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082625    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:14:49.087731 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082627    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:14:49.087731 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082630    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:14:49.087731 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082633    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:14:49.087731 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082638    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:14:49.087731 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082640    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:14:49.087731 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082643    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:14:49.087731 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082645    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:14:49.087731 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082648    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:14:49.087731 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082650    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:14:49.087731 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082653    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:14:49.087731 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.082655    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:14:49.087731 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082733    2583 flags.go:64] FLAG: --address="0.0.0.0"
Jul 02 13:14:49.087731 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082740    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 02 13:14:49.087731 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082746    2583 flags.go:64] FLAG: --anonymous-auth="true"
Jul 02 13:14:49.087731 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082751    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 02 13:14:49.087731 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082755    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 02 13:14:49.087731 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082758    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 02 13:14:49.088212 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082763    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 02 13:14:49.088212 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082767    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 02 13:14:49.088212 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082770    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 02 13:14:49.088212 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082773    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 02 13:14:49.088212 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082777    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 02 13:14:49.088212 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082780    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 02 13:14:49.088212 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082783    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 02 13:14:49.088212 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082786    2583 flags.go:64] FLAG: --cgroup-root=""
Jul 02 13:14:49.088212 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082788    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 02 13:14:49.088212 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082792    2583 flags.go:64] FLAG: --client-ca-file=""
Jul 02 13:14:49.088212 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082802    2583 flags.go:64] FLAG: --cloud-provider="external"
Jul 02 13:14:49.088212 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082806    2583 flags.go:64] FLAG: --cluster-dns="[]"
Jul 02 13:14:49.088212 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082810    2583 flags.go:64] FLAG: --cluster-domain=""
Jul 02 13:14:49.088212 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082813    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 02 13:14:49.088212 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082816    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 02 13:14:49.088212 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082819    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 02 13:14:49.088212 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082823    2583 flags.go:64] FLAG: --container-log-max-files="5"
Jul 02 13:14:49.088212 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082831    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 02 13:14:49.088212 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082835    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 02 13:14:49.088212 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082838    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 02 13:14:49.088212 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082841    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 02 13:14:49.088212 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082844    2583 flags.go:64] FLAG: --contention-profiling="false"
Jul 02 13:14:49.088212 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082848    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 02 13:14:49.088212 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082851    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 02 13:14:49.088862 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082854    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 02 13:14:49.088862 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082856    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 02 13:14:49.088862 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082861    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 02 13:14:49.088862 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082863    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 02 13:14:49.088862 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082866    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 02 13:14:49.088862 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082869    2583 flags.go:64] FLAG: --enable-load-reader="false"
Jul 02 13:14:49.088862 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082872    2583 flags.go:64] FLAG: --enable-server="true"
Jul 02 13:14:49.088862 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082875    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 02 13:14:49.088862 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082878    2583 flags.go:64] FLAG: --event-burst="100"
Jul 02 13:14:49.088862 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082881    2583 flags.go:64] FLAG: --event-qps="50"
Jul 02 13:14:49.088862 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082884    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 02 13:14:49.088862 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082887    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 02 13:14:49.088862 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082890    2583 flags.go:64] FLAG: --eviction-hard=""
Jul 02 13:14:49.088862 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082893    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 02 13:14:49.088862 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082896    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 02 13:14:49.088862 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082899    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 02 13:14:49.088862 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082902    2583 flags.go:64] FLAG: --eviction-soft=""
Jul 02 13:14:49.088862 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082905    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 02 13:14:49.088862 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082908    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 02 13:14:49.088862 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082914    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 02 13:14:49.088862 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082917    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 02 13:14:49.088862 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082920    2583 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 02 13:14:49.088862 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082923    2583 flags.go:64] FLAG: --fail-swap-on="true"
Jul 02 13:14:49.088862 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082926    2583 flags.go:64] FLAG: --feature-gates=""
Jul 02 13:14:49.088862 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082929    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 02 13:14:49.089497 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082932    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 02 13:14:49.089497 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082936    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 02 13:14:49.089497 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082939    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 02 13:14:49.089497 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082942    2583 flags.go:64] FLAG: --healthz-port="10248"
Jul 02 13:14:49.089497 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082945    2583 flags.go:64] FLAG: --help="false"
Jul 02 13:14:49.089497 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082948    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-135-143.ec2.internal"
Jul 02 13:14:49.089497 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082951    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 02 13:14:49.089497 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082954    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 02 13:14:49.089497 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082956    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 02 13:14:49.089497 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082959    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 02 13:14:49.089497 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082963    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 02 13:14:49.089497 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082966    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 02 13:14:49.089497 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082969    2583 flags.go:64] FLAG: --image-service-endpoint=""
Jul 02 13:14:49.089497 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082971    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 02 13:14:49.089497 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082974    2583 flags.go:64] FLAG: --kube-api-burst="100"
Jul 02 13:14:49.089497 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082977    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 02 13:14:49.089497 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082980    2583 flags.go:64] FLAG: --kube-api-qps="50"
Jul 02 13:14:49.089497 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082983    2583 flags.go:64] FLAG: --kube-reserved=""
Jul 02 13:14:49.089497 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082986    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 02 13:14:49.089497 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082988    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 02 13:14:49.089497 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082991    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 02 13:14:49.089497 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082994    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 02 13:14:49.089497 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.082997    2583 flags.go:64] FLAG: --lock-file=""
Jul 02 13:14:49.089497 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083000    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 02 13:14:49.090075 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083003    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 02 13:14:49.090075 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083005    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 02 13:14:49.090075 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083012    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 02 13:14:49.090075 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083016    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 02 13:14:49.090075 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083019    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 02 13:14:49.090075 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083021    2583 flags.go:64] FLAG: --logging-format="text"
Jul 02 13:14:49.090075 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083025    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 02 13:14:49.090075 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083028    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 02 13:14:49.090075 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083031    2583 flags.go:64] FLAG: --manifest-url=""
Jul 02 13:14:49.090075 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083034    2583 flags.go:64] FLAG: --manifest-url-header=""
Jul 02 13:14:49.090075 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083039    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 02 13:14:49.090075 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083042    2583 flags.go:64] FLAG: --max-open-files="1000000"
Jul 02 13:14:49.090075 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083046    2583 flags.go:64] FLAG: --max-pods="110"
Jul 02 13:14:49.090075 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083050    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 02 13:14:49.090075 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083052    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 02 13:14:49.090075 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083055    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 02 13:14:49.090075 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083058    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 02 13:14:49.090075 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083061    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 02 13:14:49.090075 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083064    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 02 13:14:49.090075 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083067    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 02 13:14:49.090075 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083075    2583 flags.go:64] FLAG: --node-status-max-images="50"
Jul 02 13:14:49.090075 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083078    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 02 13:14:49.090075 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083081    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 02 13:14:49.090075 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083084    2583 flags.go:64] FLAG: --pod-cidr=""
Jul 02 13:14:49.090688 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083086    2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04"
Jul 02 13:14:49.090688 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083091    2583 flags.go:64] FLAG: --pod-manifest-path=""
Jul 02 13:14:49.090688 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083094    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 02 13:14:49.090688 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083097    2583 flags.go:64] FLAG: --pods-per-core="0"
Jul 02 13:14:49.090688 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083100    2583 flags.go:64] FLAG: --port="10250"
Jul 02 13:14:49.090688 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083103    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 02 13:14:49.090688 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083106    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ae873f5be7c1900b"
Jul 02 13:14:49.090688 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083109    2583 flags.go:64] FLAG: --qos-reserved=""
Jul 02 13:14:49.090688 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083112    2583 flags.go:64] FLAG: --read-only-port="10255"
Jul 02 13:14:49.090688 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083115    2583 flags.go:64] FLAG: --register-node="true"
Jul 02 13:14:49.090688 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083118    2583 flags.go:64] FLAG: --register-with-taints=""
Jul 02 13:14:49.090688 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083123    2583 flags.go:64] FLAG: --registry-burst="10"
Jul 02 13:14:49.090688 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083126    2583 flags.go:64] FLAG: --registry-qps="5"
Jul 02 13:14:49.090688 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083129    2583 flags.go:64] FLAG: --reserved-cpus=""
Jul 02 13:14:49.090688 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083132    2583 flags.go:64] FLAG: --reserved-memory=""
Jul 02 13:14:49.090688 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083136    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 02 13:14:49.090688 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083139    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 02 13:14:49.090688 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083142    2583 flags.go:64] FLAG: --rotate-certificates="false"
Jul 02 13:14:49.090688 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083145    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 02 13:14:49.090688 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083147    2583 flags.go:64] FLAG: --runonce="false"
Jul 02 13:14:49.090688 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083150    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 02 13:14:49.090688 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083153    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 02 13:14:49.090688 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083156    2583 flags.go:64] FLAG: --seccomp-default="false"
Jul 02 13:14:49.090688 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083159    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 02 13:14:49.090688 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083162    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 02 13:14:49.091289 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083165    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 02 13:14:49.091289 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083168    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 02 13:14:49.091289 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083171    2583 flags.go:64] FLAG: --storage-driver-password="root"
Jul 02 13:14:49.091289 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083174    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 02 13:14:49.091289 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083177    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 02 13:14:49.091289 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083180    2583 flags.go:64] FLAG: --storage-driver-user="root"
Jul 02 13:14:49.091289 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083182    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 02 13:14:49.091289 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083185    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 02 13:14:49.091289 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083188    2583 flags.go:64] FLAG: --system-cgroups=""
Jul 02 13:14:49.091289 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083191    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 02 13:14:49.091289 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083196    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 02 13:14:49.091289 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083199    2583 flags.go:64] FLAG: --tls-cert-file=""
Jul 02 13:14:49.091289 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083202    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 02 13:14:49.091289 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083205    2583 flags.go:64] FLAG: --tls-min-version=""
Jul 02 13:14:49.091289 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083208    2583 flags.go:64] FLAG: --tls-private-key-file=""
Jul 02 13:14:49.091289 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083211    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 02 13:14:49.091289 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083213    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 02 13:14:49.091289 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083216    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 02 13:14:49.091289 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083220    2583 flags.go:64] FLAG: --v="2"
Jul 02 13:14:49.091289 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083224    2583 flags.go:64] FLAG: --version="false"
Jul 02 13:14:49.091289 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083229    2583 flags.go:64] FLAG: --vmodule=""
Jul 02 13:14:49.091289 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083233    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 02 13:14:49.091289 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.083236    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 02 13:14:49.091289 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083347    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:14:49.091289 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083352    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:14:49.091903 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083355    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:14:49.091903 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083358    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:14:49.091903 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083360    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:14:49.091903 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083364    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:14:49.091903 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083368    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:14:49.091903 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083371    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:14:49.091903 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083374    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:14:49.091903 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083376    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:14:49.091903 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083379    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:14:49.091903 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083382    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:14:49.091903 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083400    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:14:49.091903 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083404    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:14:49.091903 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083409    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:14:49.091903 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083412    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:14:49.091903 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083415    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:14:49.091903 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083417    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:14:49.091903 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083420    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:14:49.091903 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083424    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:14:49.091903 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083426    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:14:49.092488 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083429    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:14:49.092488 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083432    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:14:49.092488 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083434    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:14:49.092488 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083437    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:14:49.092488 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083441    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:14:49.092488 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083444    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:14:49.092488 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083449    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:14:49.092488 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083452    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:14:49.092488 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083455    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:14:49.092488 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083458    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:14:49.092488 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083460    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:14:49.092488 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083463    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:14:49.092488 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083467    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:14:49.092488 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083470    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:14:49.092488 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083472    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:14:49.092488 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083475    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:14:49.092488 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083478    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:14:49.092488 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083480    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:14:49.092488 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083483    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:14:49.092963 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083485    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:14:49.092963 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083488    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:14:49.092963 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083490    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:14:49.092963 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083493    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:14:49.092963 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083496    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:14:49.092963 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083498    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:14:49.092963 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083500    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:14:49.092963 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083503    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:14:49.092963 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083505    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:14:49.092963 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083508    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:14:49.092963 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083511    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:14:49.092963 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083513    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:14:49.092963 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083516    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:14:49.092963 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083518    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:14:49.092963 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083520    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:14:49.092963 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083523    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:14:49.092963 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083525    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:14:49.092963 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083528    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:14:49.092963 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083531    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:14:49.092963 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083535    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:14:49.093487 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083537    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:14:49.093487 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083540    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:14:49.093487 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083542    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:14:49.093487 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083546    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:14:49.093487 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083549    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:14:49.093487 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083554    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:14:49.093487 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083557    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:14:49.093487 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083559    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:14:49.093487 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083562    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:14:49.093487 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083564    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:14:49.093487 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083567    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:14:49.093487 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083569    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:14:49.093487 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083572    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:14:49.093487 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083574    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:14:49.093487 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083577    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:14:49.093487 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083579    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:14:49.093487 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083583    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:14:49.093487 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083587    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:14:49.093487 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083589    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:14:49.093487 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083592    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:14:49.093978 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083594    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:14:49.093978 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083597    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:14:49.093978 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083599    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:14:49.093978 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083602    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:14:49.093978 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083605    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:14:49.093978 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083607    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:14:49.093978 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083610    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:14:49.093978 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083613    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:14:49.093978 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083615    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:14:49.093978 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083618    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:14:49.093978 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083620    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:14:49.093978 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083624    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:14:49.093978 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083627    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:14:49.093978 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083629    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:14:49.093978 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083632    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:14:49.093978 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083634    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:14:49.093978 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083638    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:14:49.093978 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083642    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:14:49.093978 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.083645    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:14:49.094451 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.084545    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 13:14:49.094451 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.091913    2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 02 13:14:49.094451 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.091930    2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 02 13:14:49.094451 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.091951    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 02 13:14:49.094451 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.091958    2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 02 13:14:49.094451 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092017    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:14:49.094451 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092022    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:14:49.094451 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092025    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:14:49.094451 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092028    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:14:49.094451 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092032    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:14:49.094451 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092035    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:14:49.094451 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092038    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:14:49.094451 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092041    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:14:49.094451 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092044    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:14:49.094451 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092046    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:14:49.094822 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092049    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:14:49.094822 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092052    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:14:49.094822 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092054    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:14:49.094822 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092057    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:14:49.094822 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092059    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:14:49.094822 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092062    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:14:49.094822 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092065    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:14:49.094822 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092067    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:14:49.094822 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092070    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:14:49.094822 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092073    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:14:49.094822 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092075    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:14:49.094822 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092078    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:14:49.094822 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092081    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:14:49.094822 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092084    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:14:49.094822 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092087    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:14:49.094822 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092090    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:14:49.094822 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092092    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:14:49.094822 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092095    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:14:49.094822 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092098    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:14:49.094822 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092101    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:14:49.095309 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092104    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:14:49.095309 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092107    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:14:49.095309 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092111    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:14:49.095309 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092115    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:14:49.095309 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092117    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:14:49.095309 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092120    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:14:49.095309 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092123    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:14:49.095309 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092125    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:14:49.095309 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092128    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:14:49.095309 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092132    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:14:49.095309 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092136    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:14:49.095309 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092139    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:14:49.095309 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092142    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:14:49.095309 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092145    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:14:49.095309 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092148    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:14:49.095309 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092150    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:14:49.095309 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092153    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:14:49.095309 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092156    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:14:49.095309 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092159    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:14:49.095804 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092162    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:14:49.095804 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092165    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:14:49.095804 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092167    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:14:49.095804 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092170    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:14:49.095804 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092173    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:14:49.095804 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092176    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:14:49.095804 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092179    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:14:49.095804 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092181    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:14:49.095804 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092184    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:14:49.095804 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092187    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:14:49.095804 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092189    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:14:49.095804 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092192    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:14:49.095804 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092195    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:14:49.095804 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092198    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:14:49.095804 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092201    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:14:49.095804 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092204    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:14:49.095804 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092206    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:14:49.095804 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092209    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:14:49.095804 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092213    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:14:49.096260 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092216    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:14:49.096260 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092219    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:14:49.096260 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092222    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:14:49.096260 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092225    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:14:49.096260 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092227    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:14:49.096260 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092229    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:14:49.096260 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092232    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:14:49.096260 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092235    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:14:49.096260 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092238    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:14:49.096260 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092240    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:14:49.096260 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092243    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:14:49.096260 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092246    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:14:49.096260 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092248    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:14:49.096260 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092251    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:14:49.096260 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092253    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:14:49.096260 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092256    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:14:49.096260 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092259    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:14:49.096260 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092261    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:14:49.096260 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092264    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:14:49.096826 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092267    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:14:49.096826 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092269    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:14:49.096826 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092272    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:14:49.096826 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092279    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:14:49.096826 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092283    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:14:49.096826 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092286    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:14:49.096826 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092289    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:14:49.096826 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092292    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:14:49.096826 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092294    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:14:49.096826 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092297    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:14:49.096826 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092300    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:14:49.096826 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092302    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:14:49.096826 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.092307    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 13:14:49.096826 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092440    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:14:49.096826 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092446    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:14:49.097189 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092450    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:14:49.097189 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092453    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:14:49.097189 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092456    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:14:49.097189 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092459    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:14:49.097189 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092461    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:14:49.097189 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092464    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:14:49.097189 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092466    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:14:49.097189 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092469    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:14:49.097189 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092471    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:14:49.097189 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092474    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:14:49.097189 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092477    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:14:49.097189 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092480    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:14:49.097189 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092483    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:14:49.097189 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092485    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:14:49.097189 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092488    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:14:49.097189 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092491    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:14:49.097189 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092493    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:14:49.097189 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092496    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:14:49.097189 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092499    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:14:49.097189 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092503    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:14:49.097687 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092505    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:14:49.097687 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092508    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:14:49.097687 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092512    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:14:49.097687 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092515    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:14:49.097687 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092518    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:14:49.097687 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092521    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:14:49.097687 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092524    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:14:49.097687 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092526    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:14:49.097687 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092530    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:14:49.097687 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092533    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:14:49.097687 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092536    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:14:49.097687 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092540    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:14:49.097687 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092544    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:14:49.097687 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092547    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:14:49.097687 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092550    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:14:49.097687 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092552    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:14:49.097687 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092555    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:14:49.097687 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092558    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:14:49.097687 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092561    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:14:49.098145 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092564    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:14:49.098145 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092566    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:14:49.098145 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092569    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:14:49.098145 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092571    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:14:49.098145 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092574    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:14:49.098145 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092576    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:14:49.098145 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092579    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:14:49.098145 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092582    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:14:49.098145 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092584    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:14:49.098145 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092587    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:14:49.098145 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092590    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:14:49.098145 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092592    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:14:49.098145 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092595    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:14:49.098145 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092598    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:14:49.098145 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092600    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:14:49.098145 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092604    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:14:49.098145 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092606    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:14:49.098145 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092609    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:14:49.098145 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092612    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:14:49.098145 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092614    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:14:49.098720 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092617    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:14:49.098720 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092620    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:14:49.098720 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092622    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:14:49.098720 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092625    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:14:49.098720 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092627    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:14:49.098720 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092629    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:14:49.098720 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092632    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:14:49.098720 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092635    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:14:49.098720 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092637    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:14:49.098720 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092640    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:14:49.098720 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092642    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:14:49.098720 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092645    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:14:49.098720 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092648    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:14:49.098720 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092650    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:14:49.098720 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092652    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:14:49.098720 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092655    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:14:49.098720 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092658    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:14:49.098720 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092660    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:14:49.098720 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092663    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:14:49.098720 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092665    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:14:49.099193 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092668    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:14:49.099193 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092670    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:14:49.099193 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092673    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:14:49.099193 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092675    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:14:49.099193 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092678    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:14:49.099193 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092681    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:14:49.099193 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092683    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:14:49.099193 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092686    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:14:49.099193 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092689    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:14:49.099193 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092691    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:14:49.099193 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092694    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:14:49.099193 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092696    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:14:49.099193 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092699    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:14:49.099193 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092701    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:14:49.099193 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092704    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:14:49.099193 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092706    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:14:49.099193 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092709    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:14:49.099193 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:49.092711    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:14:49.099634 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.092716    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 13:14:49.099634 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.092838    2583 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 02 13:14:49.099634 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.094841    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 02 13:14:49.099634 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.096341    2583 server.go:1019] "Starting client certificate rotation"
Jul 02 13:14:49.099634 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.096439    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 13:14:49.099634 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.096482    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 13:14:49.099880 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.099864    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 13:14:49.104472 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.104418    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 13:14:49.113518 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.113499    2583 log.go:25] "Validated CRI v1 runtime API"
Jul 02 13:14:49.116837 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.116814    2583 log.go:25] "Validated CRI v1 image API"
Jul 02 13:14:49.117686 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.117670    2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 02 13:14:49.122194 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.122170    2583 fs.go:135] Filesystem UUIDs: map[4248072e-6f35-472b-8518-a11f4f844243:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a8372602-584b-48cb-af91-b28ef14a8f6b:/dev/nvme0n1p3]
Jul 02 13:14:49.122278 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.122191    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 02 13:14:49.128014 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.127894    2583 manager.go:216] Machine: {Timestamp:2026-07-02 13:14:49.126555062 +0000 UTC m=+0.354466227 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3092478 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d66cd518d90bee5d00088b4b96088 SystemUUID:ec2d66cd-518d-90be-e5d0-0088b4b96088 BootID:a2eae737-eef0-4673-8d8f-76ade3e79b59 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:25:e1:7b:8d:1f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:25:e1:7b:8d:1f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:dc:b0:09:3f:43 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 02 13:14:49.128014 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.128007    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 02 13:14:49.128157 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.128145    2583 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 02 13:14:49.130644 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.130618    2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 02 13:14:49.130781 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.130648    2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-143.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 02 13:14:49.131362 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.131352    2583 topology_manager.go:138] "Creating topology manager with none policy"
Jul 02 13:14:49.131407 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.131365    2583 container_manager_linux.go:309] "Creating device plugin manager"
Jul 02 13:14:49.131407 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.131373    2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 13:14:49.132089 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.132078    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 13:14:49.132896 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.132886    2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 02 13:14:49.135927 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.135915    2583 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 13:14:49.136074 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.136064    2583 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 02 13:14:49.136726 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.136716    2583 kubelet.go:486] "Attempting to sync node with API server"
Jul 02 13:14:49.136756 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.136732    2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 02 13:14:49.136783 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.136759    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 02 13:14:49.136783 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.136775    2583 kubelet.go:398] "Adding apiserver pod source"
Jul 02 13:14:49.136843 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.136788    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 02 13:14:49.137212 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.137201    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 13:14:49.137255 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.137219    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 13:14:49.139839 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.139823    2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 02 13:14:49.141409 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.141379    2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 02 13:14:49.141474 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.141426    2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 02 13:14:49.142516 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.142504    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 02 13:14:49.142551 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.142522    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 02 13:14:49.142551 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.142528    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 02 13:14:49.142551 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.142534    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 02 13:14:49.142551 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.142550    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 02 13:14:49.142676 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.142556    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 02 13:14:49.142676 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.142565    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 02 13:14:49.142676 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.142571    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 02 13:14:49.142676 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.142578    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 02 13:14:49.142676 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.142583    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 02 13:14:49.142676 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.142599    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 02 13:14:49.142676 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.142604    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 02 13:14:49.143945 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.143933    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 02 13:14:49.143978 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.143950    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 02 13:14:49.147262 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.147248    2583 server.go:1268] "Started kubelet"
Jul 02 13:14:49.147469 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.147411    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 02 13:14:49.147596 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.147493    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 02 13:14:49.147596 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.147521    2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 02 13:14:49.148128 ip-10-0-135-143 systemd[1]: Started Kubernetes Kubelet.
Jul 02 13:14:49.148727 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.148685    2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 02 13:14:49.149120 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.149106    2583 server.go:310] "Adding debug handlers to kubelet server"
Jul 02 13:14:49.151274 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.151255    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-143.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 02 13:14:49.151370 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:49.151296    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-143.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 02 13:14:49.151370 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:49.151349    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 02 13:14:49.152920 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:49.152900    2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 02 13:14:49.155359 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.155343    2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 02 13:14:49.155608 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.155347    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 02 13:14:49.155608 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:49.155447    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Jul 02 13:14:49.156150 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.156127    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 02 13:14:49.156227 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.156128    2583 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 02 13:14:49.156227 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.156168    2583 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 02 13:14:49.156304 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.156284    2583 reconstruct.go:97] "Volume reconstruction finished"
Jul 02 13:14:49.156304 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.156296    2583 reconciler.go:29] "Reconciler: start to sync state"
Jul 02 13:14:49.157525 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.157507    2583 factory.go:55] Registering systemd factory
Jul 02 13:14:49.157525 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.157526    2583 factory.go:223] Registration of the systemd container factory successfully
Jul 02 13:14:49.158029 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.157822    2583 factory.go:153] Registering CRI-O factory
Jul 02 13:14:49.158029 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.157841    2583 factory.go:223] Registration of the crio container factory successfully
Jul 02 13:14:49.158029 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.157899    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 02 13:14:49.158029 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.157922    2583 factory.go:103] Registering Raw factory
Jul 02 13:14:49.158029 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.158024    2583 manager.go:1196] Started watching for new ooms in manager
Jul 02 13:14:49.158903 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:49.158866    2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-143.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 02 13:14:49.159033 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:49.158553    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 02 13:14:49.159683 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.159665    2583 manager.go:349] Starting recovery of all containers
Jul 02 13:14:49.161628 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:49.159344    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-143.ec2.internal.18be7b50bad1f3e4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-143.ec2.internal,UID:ip-10-0-135-143.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-143.ec2.internal,},FirstTimestamp:2026-07-02 13:14:49.147216868 +0000 UTC m=+0.375128033,LastTimestamp:2026-07-02 13:14:49.147216868 +0000 UTC m=+0.375128033,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-143.ec2.internal,}"
Jul 02 13:14:49.167511 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.167483    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 13:14:49.170970 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.170953    2583 manager.go:354] Recovery completed
Jul 02 13:14:49.175257 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.175244    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:14:49.177873 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.177851    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:14:49.177936 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.177890    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:14:49.177936 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.177900    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:14:49.178543 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.178529    2583 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 02 13:14:49.178543 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.178540    2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 02 13:14:49.178671 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.178543    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:14:49.178671 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.178556    2583 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 13:14:49.179373 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.179350    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:14:49.179455 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.179381    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:14:49.179455 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.179412    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:14:49.180025 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.180012    2583 policy_none.go:49] "None policy: Start"
Jul 02 13:14:49.180077 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.180028    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 02 13:14:49.180077 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.180038    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 02 13:14:49.181205 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.181194    2583 policy_none.go:47] "Start"
Jul 02 13:14:49.183292 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:49.183195    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-143.ec2.internal.18be7b50bca5bc0e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-143.ec2.internal,UID:ip-10-0-135-143.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-143.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-143.ec2.internal,},FirstTimestamp:2026-07-02 13:14:49.177873422 +0000 UTC m=+0.405784590,LastTimestamp:2026-07-02 13:14:49.177873422 +0000 UTC m=+0.405784590,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-143.ec2.internal,}"
Jul 02 13:14:49.206773 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:49.206586    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-143.ec2.internal.18be7b50bca60fb5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-143.ec2.internal,UID:ip-10-0-135-143.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-143.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-143.ec2.internal,},FirstTimestamp:2026-07-02 13:14:49.177894837 +0000 UTC m=+0.405806003,LastTimestamp:2026-07-02 13:14:49.177894837 +0000 UTC m=+0.405806003,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-143.ec2.internal,}"
Jul 02 13:14:49.219968 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:49.219880    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-143.ec2.internal.18be7b50bca6344a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-143.ec2.internal,UID:ip-10-0-135-143.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-143.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-143.ec2.internal,},FirstTimestamp:2026-07-02 13:14:49.177904202 +0000 UTC m=+0.405815368,LastTimestamp:2026-07-02 13:14:49.177904202 +0000 UTC m=+0.405815368,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-143.ec2.internal,}"
Jul 02 13:14:49.230836 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.220060    2583 manager.go:337] "Starting Device Plugin manager"
Jul 02 13:14:49.230836 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:49.220092    2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 02 13:14:49.230836 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.220103    2583 server.go:87] "Starting device plugin registration server"
Jul 02 13:14:49.230836 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.220423    2583 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 02 13:14:49.230836 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.220436    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 02 13:14:49.230836 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.220491    2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 02 13:14:49.230836 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.220578    2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 02 13:14:49.230836 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.220589    2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 02 13:14:49.230836 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:49.223534    2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 02 13:14:49.230836 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:49.223583    2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-143.ec2.internal\" not found"
Jul 02 13:14:49.230836 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.225982    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 02 13:14:49.230836 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.227217    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 02 13:14:49.230836 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.227252    2583 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 02 13:14:49.230836 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.227278    2583 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 02 13:14:49.230836 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:49.227347    2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 02 13:14:49.231294 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:49.231052    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Jul 02 13:14:49.231294 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:49.231037    2583 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-143.ec2.internal.18be7b50bca5bc0e\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-143.ec2.internal.18be7b50bca5bc0e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-143.ec2.internal,UID:ip-10-0-135-143.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-143.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-143.ec2.internal,},FirstTimestamp:2026-07-02 13:14:49.177873422 +0000 UTC m=+0.405784590,LastTimestamp:2026-07-02 13:14:49.179365133 +0000 UTC m=+0.407276302,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-143.ec2.internal,}"
Jul 02 13:14:49.243682 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:49.243594    2583 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-143.ec2.internal.18be7b50bca60fb5\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-143.ec2.internal.18be7b50bca60fb5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-143.ec2.internal,UID:ip-10-0-135-143.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-143.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-143.ec2.internal,},FirstTimestamp:2026-07-02 13:14:49.177894837 +0000 UTC m=+0.405806003,LastTimestamp:2026-07-02 13:14:49.179403757 +0000 UTC m=+0.407314923,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-143.ec2.internal,}"
Jul 02 13:14:49.246330 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:49.246255    2583 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-143.ec2.internal.18be7b50bca6344a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-143.ec2.internal.18be7b50bca6344a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-143.ec2.internal,UID:ip-10-0-135-143.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-143.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-143.ec2.internal,},FirstTimestamp:2026-07-02 13:14:49.177904202 +0000 UTC m=+0.405815368,LastTimestamp:2026-07-02 13:14:49.179418034 +0000 UTC m=+0.407329200,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-143.ec2.internal,}"
Jul 02 13:14:49.254370 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.254350    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zgbj5"
Jul 02 13:14:49.258149 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:49.258085    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-143.ec2.internal.18be7b50bf5aad33  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-143.ec2.internal,UID:ip-10-0-135-143.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-135-143.ec2.internal,},FirstTimestamp:2026-07-02 13:14:49.223286067 +0000 UTC m=+0.451197219,LastTimestamp:2026-07-02 13:14:49.223286067 +0000 UTC m=+0.451197219,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-143.ec2.internal,}"
Jul 02 13:14:49.265929 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.265907    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zgbj5"
Jul 02 13:14:49.320819 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.320744    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:14:49.321712 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.321694    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:14:49.321779 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.321725    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:14:49.321779 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.321734    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:14:49.321779 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.321759    2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-143.ec2.internal"
Jul 02 13:14:49.327916 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.327897    2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal"]
Jul 02 13:14:49.327969 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.327958    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:14:49.329347 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.329333    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:14:49.329408 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.329360    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:14:49.329408 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.329369    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:14:49.330665 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.330653    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:14:49.330793 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.330777    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal"
Jul 02 13:14:49.330830 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.330808    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:14:49.331370 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.331349    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:14:49.331451 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.331376    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:14:49.331451 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.331405    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:14:49.331849 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.331831    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:14:49.331893 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.331859    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:14:49.331893 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.331869    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:14:49.332831 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.332817    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal"
Jul 02 13:14:49.332872 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.332843    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:14:49.333550 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.333533    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:14:49.333619 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.333565    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:14:49.333619 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.333579    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:14:49.347772 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.347745    2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-143.ec2.internal"
Jul 02 13:14:49.347772 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:49.347773    2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-143.ec2.internal\": node \"ip-10-0-135-143.ec2.internal\" not found"
Jul 02 13:14:49.353033 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:49.353005    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-143.ec2.internal\" not found" node="ip-10-0-135-143.ec2.internal"
Jul 02 13:14:49.357935 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.357915    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ef0c8167d6f730479020bb6b4e702a8c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal\" (UID: \"ef0c8167d6f730479020bb6b4e702a8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal"
Jul 02 13:14:49.358035 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.357942    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef0c8167d6f730479020bb6b4e702a8c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal\" (UID: \"ef0c8167d6f730479020bb6b4e702a8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal"
Jul 02 13:14:49.358035 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.357960    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b72efbb5a6c47d6a170c394c0a860619-config\") pod \"kube-apiserver-proxy-ip-10-0-135-143.ec2.internal\" (UID: \"b72efbb5a6c47d6a170c394c0a860619\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal"
Jul 02 13:14:49.358189 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:49.358171    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-143.ec2.internal\" not found" node="ip-10-0-135-143.ec2.internal"
Jul 02 13:14:49.449033 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:49.449006    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Jul 02 13:14:49.458245 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.458212    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ef0c8167d6f730479020bb6b4e702a8c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal\" (UID: \"ef0c8167d6f730479020bb6b4e702a8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal"
Jul 02 13:14:49.458368 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.458251    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef0c8167d6f730479020bb6b4e702a8c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal\" (UID: \"ef0c8167d6f730479020bb6b4e702a8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal"
Jul 02 13:14:49.458368 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.458274    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b72efbb5a6c47d6a170c394c0a860619-config\") pod \"kube-apiserver-proxy-ip-10-0-135-143.ec2.internal\" (UID: \"b72efbb5a6c47d6a170c394c0a860619\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal"
Jul 02 13:14:49.458368 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.458317    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b72efbb5a6c47d6a170c394c0a860619-config\") pod \"kube-apiserver-proxy-ip-10-0-135-143.ec2.internal\" (UID: \"b72efbb5a6c47d6a170c394c0a860619\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal"
Jul 02 13:14:49.458368 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.458321    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef0c8167d6f730479020bb6b4e702a8c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal\" (UID: \"ef0c8167d6f730479020bb6b4e702a8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal"
Jul 02 13:14:49.458368 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.458321    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ef0c8167d6f730479020bb6b4e702a8c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal\" (UID: \"ef0c8167d6f730479020bb6b4e702a8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal"
Jul 02 13:14:49.549720 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:49.549687    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Jul 02 13:14:49.650668 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:49.650630    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Jul 02 13:14:49.657987 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.657964    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal"
Jul 02 13:14:49.662046 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:49.662028    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal"
Jul 02 13:14:49.751747 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:49.751716    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Jul 02 13:14:49.852326 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:49.852295    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Jul 02 13:14:49.952858 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:49.952776    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Jul 02 13:14:50.053359 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:50.053329    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Jul 02 13:14:50.095798 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:50.095767    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 02 13:14:50.153979 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:50.153944    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-143.ec2.internal\" not found"
Jul 02 13:14:50.185163 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:50.185125    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:14:50.214590 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:50.214551    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb72efbb5a6c47d6a170c394c0a860619.slice/crio-4ef2156d7f80319180b0494e0ec7ddd3398ec5b632996a8620ac5a99dc323aad WatchSource:0}: Error finding container 4ef2156d7f80319180b0494e0ec7ddd3398ec5b632996a8620ac5a99dc323aad: Status 404 returned error can't find the container with id 4ef2156d7f80319180b0494e0ec7ddd3398ec5b632996a8620ac5a99dc323aad
Jul 02 13:14:50.217967 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:50.217946    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:14:50.219990 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:50.219974    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 13:14:50.230971 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:50.230925    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal" event={"ID":"b72efbb5a6c47d6a170c394c0a860619","Type":"ContainerStarted","Data":"4ef2156d7f80319180b0494e0ec7ddd3398ec5b632996a8620ac5a99dc323aad"}
Jul 02 13:14:50.237557 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:50.237532    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef0c8167d6f730479020bb6b4e702a8c.slice/crio-9c636fe8e81f7edb9373d01aff31254c54c2661bf64901006c9cc417bf84afd3 WatchSource:0}: Error finding container 9c636fe8e81f7edb9373d01aff31254c54c2661bf64901006c9cc417bf84afd3: Status 404 returned error can't find the container with id 9c636fe8e81f7edb9373d01aff31254c54c2661bf64901006c9cc417bf84afd3
Jul 02 13:14:50.256291 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:50.256271    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal"
Jul 02 13:14:50.270606 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:50.270569    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-01 13:09:49 +0000 UTC" deadline="2027-11-30 12:55:48.710952118 +0000 UTC"
Jul 02 13:14:50.270606 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:50.270598    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12383h40m58.440357784s"
Jul 02 13:14:50.299609 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:50.299583    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 13:14:50.300494 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:50.300479    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal"
Jul 02 13:14:50.315430 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:50.315406    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 13:14:50.462664 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:50.462591    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:14:50.620252 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:50.620220    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:14:51.138132 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.138100    2583 apiserver.go:52] "Watching apiserver"
Jul 02 13:14:51.147996 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.147959    2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 02 13:14:51.148369 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.148340    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-jlkds","kube-system/konnectivity-agent-7jfsq","kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal","openshift-cluster-node-tuning-operator/tuned-6rfcq","openshift-dns/node-resolver-hzspc","openshift-image-registry/node-ca-9pvbh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal","openshift-multus/multus-additional-cni-plugins-fjbsm","openshift-network-diagnostics/network-check-target-tx24z","kube-system/global-pull-secret-syncer-9fpt5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67gtm","openshift-multus/multus-2hwvd","openshift-multus/network-metrics-daemon-xzl6q","openshift-network-operator/iptables-alerter-9m4tg"]
Jul 02 13:14:51.150152 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.150039    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:14:51.150152 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:51.150126    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fpt5" podUID="bfa11a9f-ca92-4214-8e84-0b9c48a8337c"
Jul 02 13:14:51.151288 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.151263    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:14:51.151381 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:51.151339    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzl6q" podUID="0cae18b4-8f5b-4ff7-a73f-fc668af0a663"
Jul 02 13:14:51.152618 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.152599    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9m4tg"
Jul 02 13:14:51.154280 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.154258    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.156137 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.156118    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 02 13:14:51.156137 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.156134    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hzspc"
Jul 02 13:14:51.157497 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.157406    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9pvbh"
Jul 02 13:14:51.157833 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.157813    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sb956\""
Jul 02 13:14:51.157968 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.157911    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 02 13:14:51.157968 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.157959    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 02 13:14:51.158099 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.157973    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 02 13:14:51.158891 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.158860    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:14:51.158983 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:51.158928    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx24z" podUID="159f78cc-1fe0-40ca-a0cd-92254294389c"
Jul 02 13:14:51.160339 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.160319    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fjbsm"
Jul 02 13:14:51.161947 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.161462    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 02 13:14:51.162134 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.162117    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 02 13:14:51.162539 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.162522    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-b86vc\""
Jul 02 13:14:51.162837 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.162818    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p6cw6\""
Jul 02 13:14:51.163195 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.163174    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 02 13:14:51.163267 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.163203    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 02 13:14:51.163907 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.163883    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 02 13:14:51.164186 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.164142    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-49kxc\""
Jul 02 13:14:51.164434 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.164381    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 02 13:14:51.164516 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.164502    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 02 13:14:51.165563 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.165463    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7jfsq"
Jul 02 13:14:51.167852 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.167830    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-etc-sysctl-d\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.167950 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.167864    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2lxn\" (UniqueName: \"kubernetes.io/projected/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-kube-api-access-v2lxn\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.167950 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.167886    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-sys\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.167950 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.167936    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwsp4\" (UniqueName: \"kubernetes.io/projected/a2b80dbb-5bb9-4d5e-add9-aeb0519f92c1-kube-api-access-qwsp4\") pod \"node-resolver-hzspc\" (UID: \"a2b80dbb-5bb9-4d5e-add9-aeb0519f92c1\") " pod="openshift-dns/node-resolver-hzspc"
Jul 02 13:14:51.168172 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.167964    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bfa11a9f-ca92-4214-8e84-0b9c48a8337c-kubelet-config\") pod \"global-pull-secret-syncer-9fpt5\" (UID: \"bfa11a9f-ca92-4214-8e84-0b9c48a8337c\") " pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:14:51.168172 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.167987    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dc9c602d-4c56-4fe4-b1d7-5ef367ca6382-iptables-alerter-script\") pod \"iptables-alerter-9m4tg\" (UID: \"dc9c602d-4c56-4fe4-b1d7-5ef367ca6382\") " pod="openshift-network-operator/iptables-alerter-9m4tg"
Jul 02 13:14:51.168172 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.168010    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-etc-sysctl-conf\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.168172 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.168031    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-lib-modules\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.168172 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.168055    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2b80dbb-5bb9-4d5e-add9-aeb0519f92c1-tmp-dir\") pod \"node-resolver-hzspc\" (UID: \"a2b80dbb-5bb9-4d5e-add9-aeb0519f92c1\") " pod="openshift-dns/node-resolver-hzspc"
Jul 02 13:14:51.168172 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.168076    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bfa11a9f-ca92-4214-8e84-0b9c48a8337c-dbus\") pod \"global-pull-secret-syncer-9fpt5\" (UID: \"bfa11a9f-ca92-4214-8e84-0b9c48a8337c\") " pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:14:51.168172 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.168101    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-etc-modprobe-d\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.168172 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.168124    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cae18b4-8f5b-4ff7-a73f-fc668af0a663-metrics-certs\") pod \"network-metrics-daemon-xzl6q\" (UID: \"0cae18b4-8f5b-4ff7-a73f-fc668af0a663\") " pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:14:51.168172 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.168145    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-etc-kubernetes\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.168172 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.168167    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-tmp\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.168639 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.168189    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a2b80dbb-5bb9-4d5e-add9-aeb0519f92c1-hosts-file\") pod \"node-resolver-hzspc\" (UID: \"a2b80dbb-5bb9-4d5e-add9-aeb0519f92c1\") " pod="openshift-dns/node-resolver-hzspc"
Jul 02 13:14:51.168639 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.168210    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2cccfac-f483-4a64-8788-b78393b5a88e-host\") pod \"node-ca-9pvbh\" (UID: \"b2cccfac-f483-4a64-8788-b78393b5a88e\") " pod="openshift-image-registry/node-ca-9pvbh"
Jul 02 13:14:51.168639 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.168234    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bfa11a9f-ca92-4214-8e84-0b9c48a8337c-original-pull-secret\") pod \"global-pull-secret-syncer-9fpt5\" (UID: \"bfa11a9f-ca92-4214-8e84-0b9c48a8337c\") " pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:14:51.168639 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.168258    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-run\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.168639 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.168280    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkczs\" (UniqueName: \"kubernetes.io/projected/0cae18b4-8f5b-4ff7-a73f-fc668af0a663-kube-api-access-wkczs\") pod \"network-metrics-daemon-xzl6q\" (UID: \"0cae18b4-8f5b-4ff7-a73f-fc668af0a663\") " pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:14:51.168639 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.168303    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-etc-sysconfig\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.168639 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.168352    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-etc-systemd\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.168639 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.168401    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-var-lib-kubelet\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.168639 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.168427    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-etc-tuned\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.168639 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.168450    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b2cccfac-f483-4a64-8788-b78393b5a88e-serviceca\") pod \"node-ca-9pvbh\" (UID: \"b2cccfac-f483-4a64-8788-b78393b5a88e\") " pod="openshift-image-registry/node-ca-9pvbh"
Jul 02 13:14:51.168639 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.168474    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vlsx\" (UniqueName: \"kubernetes.io/projected/dc9c602d-4c56-4fe4-b1d7-5ef367ca6382-kube-api-access-2vlsx\") pod \"iptables-alerter-9m4tg\" (UID: \"dc9c602d-4c56-4fe4-b1d7-5ef367ca6382\") " pod="openshift-network-operator/iptables-alerter-9m4tg"
Jul 02 13:14:51.168639 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.168499    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-host\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.168639 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.168546    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h9k7\" (UniqueName: \"kubernetes.io/projected/b2cccfac-f483-4a64-8788-b78393b5a88e-kube-api-access-5h9k7\") pod \"node-ca-9pvbh\" (UID: \"b2cccfac-f483-4a64-8788-b78393b5a88e\") " pod="openshift-image-registry/node-ca-9pvbh"
Jul 02 13:14:51.168639 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.168576    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc9c602d-4c56-4fe4-b1d7-5ef367ca6382-host-slash\") pod \"iptables-alerter-9m4tg\" (UID: \"dc9c602d-4c56-4fe4-b1d7-5ef367ca6382\") " pod="openshift-network-operator/iptables-alerter-9m4tg"
Jul 02 13:14:51.169102 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.168713    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.170285 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.170267    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67gtm"
Jul 02 13:14:51.172412 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.172117    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.177008 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.176965    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 02 13:14:51.177110 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.177009    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 02 13:14:51.177479 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.177276    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 02 13:14:51.177479 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.177283    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 02 13:14:51.177479 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.177300    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 02 13:14:51.177479 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.177333    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 02 13:14:51.177479 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.177473    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 02 13:14:51.178193 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.178176    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 02 13:14:51.178193 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.178187    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 02 13:14:51.178595 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.178478    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 02 13:14:51.178595 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.178492    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 02 13:14:51.178595 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.178481    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 02 13:14:51.179276 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.179258    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 02 13:14:51.179594 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.179571    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nfq5w\""
Jul 02 13:14:51.179701 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.179338    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 02 13:14:51.180077 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.179499    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7rf9g\""
Jul 02 13:14:51.180077 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.179690    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 02 13:14:51.180077 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.180004    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 02 13:14:51.180077 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.179286    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 02 13:14:51.180299 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.180126    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-47nlq\""
Jul 02 13:14:51.180299 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.180279    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-z4w8t\""
Jul 02 13:14:51.180427 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.180317    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vvjkt\""
Jul 02 13:14:51.181829 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.181790    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 13:14:51.203119 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.203098    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ph4zq"
Jul 02 13:14:51.216507 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.216468    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ph4zq"
Jul 02 13:14:51.233109 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.233074    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal" event={"ID":"ef0c8167d6f730479020bb6b4e702a8c","Type":"ContainerStarted","Data":"9c636fe8e81f7edb9373d01aff31254c54c2661bf64901006c9cc417bf84afd3"}
Jul 02 13:14:51.257701 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.257673    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 02 13:14:51.269699 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.269664    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vlsx\" (UniqueName: \"kubernetes.io/projected/dc9c602d-4c56-4fe4-b1d7-5ef367ca6382-kube-api-access-2vlsx\") pod \"iptables-alerter-9m4tg\" (UID: \"dc9c602d-4c56-4fe4-b1d7-5ef367ca6382\") " pod="openshift-network-operator/iptables-alerter-9m4tg"
Jul 02 13:14:51.269699 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.269704    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-host\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.269919 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.269725    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h9k7\" (UniqueName: \"kubernetes.io/projected/b2cccfac-f483-4a64-8788-b78393b5a88e-kube-api-access-5h9k7\") pod \"node-ca-9pvbh\" (UID: \"b2cccfac-f483-4a64-8788-b78393b5a88e\") " pod="openshift-image-registry/node-ca-9pvbh"
Jul 02 13:14:51.269919 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.269750    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-etc-sysctl-d\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.269919 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.269805    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-host\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.269919 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.269879    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2lxn\" (UniqueName: \"kubernetes.io/projected/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-kube-api-access-v2lxn\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.269919 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.269900    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-etc-sysctl-d\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.269919 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.269910    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4c5abfcf-f610-4f74-9222-a321d08c0a1b-agent-certs\") pod \"konnectivity-agent-7jfsq\" (UID: \"4c5abfcf-f610-4f74-9222-a321d08c0a1b\") " pod="kube-system/konnectivity-agent-7jfsq"
Jul 02 13:14:51.270181 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.269934    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-hostroot\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.270181 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.269961    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/249b4a4b-a5eb-4bc1-8311-1c330010730d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fjbsm\" (UID: \"249b4a4b-a5eb-4bc1-8311-1c330010730d\") " pod="openshift-multus/multus-additional-cni-plugins-fjbsm"
Jul 02 13:14:51.270181 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.269998    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.270181 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.270025    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-sys\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.270181 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.270058    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwsp4\" (UniqueName: \"kubernetes.io/projected/a2b80dbb-5bb9-4d5e-add9-aeb0519f92c1-kube-api-access-qwsp4\") pod \"node-resolver-hzspc\" (UID: \"a2b80dbb-5bb9-4d5e-add9-aeb0519f92c1\") " pod="openshift-dns/node-resolver-hzspc"
Jul 02 13:14:51.270181 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.270081    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-host-run-k8s-cni-cncf-io\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.270181 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.270105    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-sys\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.270181 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.270113    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-host-var-lib-cni-bin\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.270181 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.270137    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/249b4a4b-a5eb-4bc1-8311-1c330010730d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fjbsm\" (UID: \"249b4a4b-a5eb-4bc1-8311-1c330010730d\") " pod="openshift-multus/multus-additional-cni-plugins-fjbsm"
Jul 02 13:14:51.270668 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.270244    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-host-run-ovn-kubernetes\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.270668 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.270293    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dc9c602d-4c56-4fe4-b1d7-5ef367ca6382-iptables-alerter-script\") pod \"iptables-alerter-9m4tg\" (UID: \"dc9c602d-4c56-4fe4-b1d7-5ef367ca6382\") " pod="openshift-network-operator/iptables-alerter-9m4tg"
Jul 02 13:14:51.270668 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.270340    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-system-cni-dir\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.270668 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.270367    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-cni-binary-copy\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.270668 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.270407    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-host-var-lib-kubelet\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.270668 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.270439    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-systemd-units\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.270668 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.270497    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-host-run-netns\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.270668 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.270521    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a2b80dbb-5bb9-4d5e-add9-aeb0519f92c1-hosts-file\") pod \"node-resolver-hzspc\" (UID: \"a2b80dbb-5bb9-4d5e-add9-aeb0519f92c1\") " pod="openshift-dns/node-resolver-hzspc"
Jul 02 13:14:51.270668 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.270563    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-os-release\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.270668 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.270598    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/249b4a4b-a5eb-4bc1-8311-1c330010730d-os-release\") pod \"multus-additional-cni-plugins-fjbsm\" (UID: \"249b4a4b-a5eb-4bc1-8311-1c330010730d\") " pod="openshift-multus/multus-additional-cni-plugins-fjbsm"
Jul 02 13:14:51.270668 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.270624    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bfa11a9f-ca92-4214-8e84-0b9c48a8337c-original-pull-secret\") pod \"global-pull-secret-syncer-9fpt5\" (UID: \"bfa11a9f-ca92-4214-8e84-0b9c48a8337c\") " pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:14:51.270668 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.270655    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-run\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.271220 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.270683    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkczs\" (UniqueName: \"kubernetes.io/projected/0cae18b4-8f5b-4ff7-a73f-fc668af0a663-kube-api-access-wkczs\") pod \"network-metrics-daemon-xzl6q\" (UID: \"0cae18b4-8f5b-4ff7-a73f-fc668af0a663\") " pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:14:51.271220 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.270711    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-etc-sysconfig\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.271220 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.270739    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b2cccfac-f483-4a64-8788-b78393b5a88e-serviceca\") pod \"node-ca-9pvbh\" (UID: \"b2cccfac-f483-4a64-8788-b78393b5a88e\") " pod="openshift-image-registry/node-ca-9pvbh"
Jul 02 13:14:51.271220 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.270766    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-host-var-lib-cni-multus\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.271220 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.270793    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-multus-conf-dir\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.271220 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.270821    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncsvd\" (UniqueName: \"kubernetes.io/projected/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-kube-api-access-ncsvd\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.271220 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.270847    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/249b4a4b-a5eb-4bc1-8311-1c330010730d-system-cni-dir\") pod \"multus-additional-cni-plugins-fjbsm\" (UID: \"249b4a4b-a5eb-4bc1-8311-1c330010730d\") " pod="openshift-multus/multus-additional-cni-plugins-fjbsm"
Jul 02 13:14:51.271220 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.270875    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/249b4a4b-a5eb-4bc1-8311-1c330010730d-cnibin\") pod \"multus-additional-cni-plugins-fjbsm\" (UID: \"249b4a4b-a5eb-4bc1-8311-1c330010730d\") " pod="openshift-multus/multus-additional-cni-plugins-fjbsm"
Jul 02 13:14:51.271220 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.270904    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/249b4a4b-a5eb-4bc1-8311-1c330010730d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fjbsm\" (UID: \"249b4a4b-a5eb-4bc1-8311-1c330010730d\") " pod="openshift-multus/multus-additional-cni-plugins-fjbsm"
Jul 02 13:14:51.271220 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.270918    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dc9c602d-4c56-4fe4-b1d7-5ef367ca6382-iptables-alerter-script\") pod \"iptables-alerter-9m4tg\" (UID: \"dc9c602d-4c56-4fe4-b1d7-5ef367ca6382\") " pod="openshift-network-operator/iptables-alerter-9m4tg"
Jul 02 13:14:51.271220 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.270939    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-multus-daemon-config\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.271220 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.270968    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-run-ovn\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.271220 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.271012    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-node-log\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.271220 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.271039    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-host-cni-bin\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.271220 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.271061    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc9c602d-4c56-4fe4-b1d7-5ef367ca6382-host-slash\") pod \"iptables-alerter-9m4tg\" (UID: \"dc9c602d-4c56-4fe4-b1d7-5ef367ca6382\") " pod="openshift-network-operator/iptables-alerter-9m4tg"
Jul 02 13:14:51.271220 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.271079    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a2b80dbb-5bb9-4d5e-add9-aeb0519f92c1-hosts-file\") pod \"node-resolver-hzspc\" (UID: \"a2b80dbb-5bb9-4d5e-add9-aeb0519f92c1\") " pod="openshift-dns/node-resolver-hzspc"
Jul 02 13:14:51.271220 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.271105    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc9c602d-4c56-4fe4-b1d7-5ef367ca6382-host-slash\") pod \"iptables-alerter-9m4tg\" (UID: \"dc9c602d-4c56-4fe4-b1d7-5ef367ca6382\") " pod="openshift-network-operator/iptables-alerter-9m4tg"
Jul 02 13:14:51.272008 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.271116    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-run\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.272008 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:51.271128    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:51.272008 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.271161    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-run-systemd\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.272008 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:51.271201    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfa11a9f-ca92-4214-8e84-0b9c48a8337c-original-pull-secret podName:bfa11a9f-ca92-4214-8e84-0b9c48a8337c nodeName:}" failed. No retries permitted until 2026-07-02 13:14:51.771184071 +0000 UTC m=+2.999095223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bfa11a9f-ca92-4214-8e84-0b9c48a8337c-original-pull-secret") pod "global-pull-secret-syncer-9fpt5" (UID: "bfa11a9f-ca92-4214-8e84-0b9c48a8337c") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:51.272008 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.271278    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-etc-sysconfig\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.272008 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.271358    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-log-socket\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.272008 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.271409    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f0256660-1798-4dba-a311-3705cb73c9ca-env-overrides\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.272008 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.271515    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch7rl\" (UniqueName: \"kubernetes.io/projected/f0256660-1798-4dba-a311-3705cb73c9ca-kube-api-access-ch7rl\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.272008 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.271558    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2-registration-dir\") pod \"aws-ebs-csi-driver-node-67gtm\" (UID: \"dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67gtm"
Jul 02 13:14:51.272008 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.271598    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4c5abfcf-f610-4f74-9222-a321d08c0a1b-konnectivity-ca\") pod \"konnectivity-agent-7jfsq\" (UID: \"4c5abfcf-f610-4f74-9222-a321d08c0a1b\") " pod="kube-system/konnectivity-agent-7jfsq"
Jul 02 13:14:51.272008 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.271631    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-etc-kubernetes\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.272008 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.271660    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-host-kubelet\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.272008 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.271684    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-host-slash\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.272008 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.271693    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b2cccfac-f483-4a64-8788-b78393b5a88e-serviceca\") pod \"node-ca-9pvbh\" (UID: \"b2cccfac-f483-4a64-8788-b78393b5a88e\") " pod="openshift-image-registry/node-ca-9pvbh"
Jul 02 13:14:51.272008 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.271710    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f0256660-1798-4dba-a311-3705cb73c9ca-ovnkube-script-lib\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.272008 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.271742    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bfa11a9f-ca92-4214-8e84-0b9c48a8337c-kubelet-config\") pod \"global-pull-secret-syncer-9fpt5\" (UID: \"bfa11a9f-ca92-4214-8e84-0b9c48a8337c\") " pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:14:51.272008 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.271774    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-etc-sysctl-conf\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.272804 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.271801    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-lib-modules\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.272804 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.271823    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2b80dbb-5bb9-4d5e-add9-aeb0519f92c1-tmp-dir\") pod \"node-resolver-hzspc\" (UID: \"a2b80dbb-5bb9-4d5e-add9-aeb0519f92c1\") " pod="openshift-dns/node-resolver-hzspc"
Jul 02 13:14:51.272804 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.271826    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bfa11a9f-ca92-4214-8e84-0b9c48a8337c-kubelet-config\") pod \"global-pull-secret-syncer-9fpt5\" (UID: \"bfa11a9f-ca92-4214-8e84-0b9c48a8337c\") " pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:14:51.272804 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.271847    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-host-run-multus-certs\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.272804 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.271867    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-run-openvswitch\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.272804 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.271884    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-host-cni-netd\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.272804 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.271904    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f0256660-1798-4dba-a311-3705cb73c9ca-ovnkube-config\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.272804 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.271935    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bfa11a9f-ca92-4214-8e84-0b9c48a8337c-dbus\") pod \"global-pull-secret-syncer-9fpt5\" (UID: \"bfa11a9f-ca92-4214-8e84-0b9c48a8337c\") " pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:14:51.272804 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.271952    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-etc-sysctl-conf\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.272804 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.271958    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-etc-modprobe-d\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.272804 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.271954    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-lib-modules\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.272804 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.271980    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-multus-cni-dir\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.272804 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.272002    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f0256660-1798-4dba-a311-3705cb73c9ca-ovn-node-metrics-cert\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.272804 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.272038    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-67gtm\" (UID: \"dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67gtm"
Jul 02 13:14:51.272804 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.272061    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-etc-modprobe-d\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.272804 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.272072    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2-socket-dir\") pod \"aws-ebs-csi-driver-node-67gtm\" (UID: \"dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67gtm"
Jul 02 13:14:51.272804 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.272079    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bfa11a9f-ca92-4214-8e84-0b9c48a8337c-dbus\") pod \"global-pull-secret-syncer-9fpt5\" (UID: \"bfa11a9f-ca92-4214-8e84-0b9c48a8337c\") " pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:14:51.273552 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.272089    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2b80dbb-5bb9-4d5e-add9-aeb0519f92c1-tmp-dir\") pod \"node-resolver-hzspc\" (UID: \"a2b80dbb-5bb9-4d5e-add9-aeb0519f92c1\") " pod="openshift-dns/node-resolver-hzspc"
Jul 02 13:14:51.273552 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.272100    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2-device-dir\") pod \"aws-ebs-csi-driver-node-67gtm\" (UID: \"dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67gtm"
Jul 02 13:14:51.273552 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.272129    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cae18b4-8f5b-4ff7-a73f-fc668af0a663-metrics-certs\") pod \"network-metrics-daemon-xzl6q\" (UID: \"0cae18b4-8f5b-4ff7-a73f-fc668af0a663\") " pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:14:51.273552 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.272165    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-etc-kubernetes\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.273552 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.272205    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-etc-kubernetes\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.273552 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:51.272236    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:51.273552 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:51.272287    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cae18b4-8f5b-4ff7-a73f-fc668af0a663-metrics-certs podName:0cae18b4-8f5b-4ff7-a73f-fc668af0a663 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:51.772270379 +0000 UTC m=+3.000181534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cae18b4-8f5b-4ff7-a73f-fc668af0a663-metrics-certs") pod "network-metrics-daemon-xzl6q" (UID: "0cae18b4-8f5b-4ff7-a73f-fc668af0a663") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:51.273552 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.272315    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-tmp\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.273552 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.272346    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2cccfac-f483-4a64-8788-b78393b5a88e-host\") pod \"node-ca-9pvbh\" (UID: \"b2cccfac-f483-4a64-8788-b78393b5a88e\") " pod="openshift-image-registry/node-ca-9pvbh"
Jul 02 13:14:51.273552 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.272373    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2r82\" (UniqueName: \"kubernetes.io/projected/159f78cc-1fe0-40ca-a0cd-92254294389c-kube-api-access-s2r82\") pod \"network-check-target-tx24z\" (UID: \"159f78cc-1fe0-40ca-a0cd-92254294389c\") " pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:14:51.273552 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.272418    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-host-run-netns\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.273552 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.272421    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2cccfac-f483-4a64-8788-b78393b5a88e-host\") pod \"node-ca-9pvbh\" (UID: \"b2cccfac-f483-4a64-8788-b78393b5a88e\") " pod="openshift-image-registry/node-ca-9pvbh"
Jul 02 13:14:51.273552 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.272454    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-etc-openvswitch\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.273552 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.272481    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fl8g\" (UniqueName: \"kubernetes.io/projected/dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2-kube-api-access-6fl8g\") pod \"aws-ebs-csi-driver-node-67gtm\" (UID: \"dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67gtm"
Jul 02 13:14:51.273552 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.272861    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-cnibin\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.273552 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.272902    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/249b4a4b-a5eb-4bc1-8311-1c330010730d-cni-binary-copy\") pod \"multus-additional-cni-plugins-fjbsm\" (UID: \"249b4a4b-a5eb-4bc1-8311-1c330010730d\") " pod="openshift-multus/multus-additional-cni-plugins-fjbsm"
Jul 02 13:14:51.273552 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.272934    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8qrj\" (UniqueName: \"kubernetes.io/projected/249b4a4b-a5eb-4bc1-8311-1c330010730d-kube-api-access-b8qrj\") pod \"multus-additional-cni-plugins-fjbsm\" (UID: \"249b4a4b-a5eb-4bc1-8311-1c330010730d\") " pod="openshift-multus/multus-additional-cni-plugins-fjbsm"
Jul 02 13:14:51.274302 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.272968    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2-sys-fs\") pod \"aws-ebs-csi-driver-node-67gtm\" (UID: \"dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67gtm"
Jul 02 13:14:51.274302 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.272998    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-etc-systemd\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.274302 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.273018    2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 02 13:14:51.274302 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.273029    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-var-lib-kubelet\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.274302 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.273060    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-etc-tuned\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.274302 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.273091    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-multus-socket-dir-parent\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.274302 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.273124    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-var-lib-openvswitch\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.274302 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.273150    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2-etc-selinux\") pod \"aws-ebs-csi-driver-node-67gtm\" (UID: \"dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67gtm"
Jul 02 13:14:51.274302 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.273608    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-etc-systemd\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.274302 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.273649    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-var-lib-kubelet\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.277135 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.277104    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-tmp\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.278582 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.278560    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-etc-tuned\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.288231 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.288192    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vlsx\" (UniqueName: \"kubernetes.io/projected/dc9c602d-4c56-4fe4-b1d7-5ef367ca6382-kube-api-access-2vlsx\") pod \"iptables-alerter-9m4tg\" (UID: \"dc9c602d-4c56-4fe4-b1d7-5ef367ca6382\") " pod="openshift-network-operator/iptables-alerter-9m4tg"
Jul 02 13:14:51.289608 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.289529    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwsp4\" (UniqueName: \"kubernetes.io/projected/a2b80dbb-5bb9-4d5e-add9-aeb0519f92c1-kube-api-access-qwsp4\") pod \"node-resolver-hzspc\" (UID: \"a2b80dbb-5bb9-4d5e-add9-aeb0519f92c1\") " pod="openshift-dns/node-resolver-hzspc"
Jul 02 13:14:51.289760 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.289637    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h9k7\" (UniqueName: \"kubernetes.io/projected/b2cccfac-f483-4a64-8788-b78393b5a88e-kube-api-access-5h9k7\") pod \"node-ca-9pvbh\" (UID: \"b2cccfac-f483-4a64-8788-b78393b5a88e\") " pod="openshift-image-registry/node-ca-9pvbh"
Jul 02 13:14:51.289760 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.289666    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2lxn\" (UniqueName: \"kubernetes.io/projected/ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1-kube-api-access-v2lxn\") pod \"tuned-6rfcq\" (UID: \"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1\") " pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.292102 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.292032    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkczs\" (UniqueName: \"kubernetes.io/projected/0cae18b4-8f5b-4ff7-a73f-fc668af0a663-kube-api-access-wkczs\") pod \"network-metrics-daemon-xzl6q\" (UID: \"0cae18b4-8f5b-4ff7-a73f-fc668af0a663\") " pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:14:51.374342 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.374308    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-host-run-netns\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.374342 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.374350    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-etc-openvswitch\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.374605 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.374377    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fl8g\" (UniqueName: \"kubernetes.io/projected/dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2-kube-api-access-6fl8g\") pod \"aws-ebs-csi-driver-node-67gtm\" (UID: \"dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67gtm"
Jul 02 13:14:51.374605 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.374420    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-cnibin\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.374605 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.374442    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/249b4a4b-a5eb-4bc1-8311-1c330010730d-cni-binary-copy\") pod \"multus-additional-cni-plugins-fjbsm\" (UID: \"249b4a4b-a5eb-4bc1-8311-1c330010730d\") " pod="openshift-multus/multus-additional-cni-plugins-fjbsm"
Jul 02 13:14:51.374605 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.374449    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-host-run-netns\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.374605 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.374473    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-etc-openvswitch\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.374605 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.374458    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8qrj\" (UniqueName: \"kubernetes.io/projected/249b4a4b-a5eb-4bc1-8311-1c330010730d-kube-api-access-b8qrj\") pod \"multus-additional-cni-plugins-fjbsm\" (UID: \"249b4a4b-a5eb-4bc1-8311-1c330010730d\") " pod="openshift-multus/multus-additional-cni-plugins-fjbsm"
Jul 02 13:14:51.374605 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.374529    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2-sys-fs\") pod \"aws-ebs-csi-driver-node-67gtm\" (UID: \"dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67gtm"
Jul 02 13:14:51.374605 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.374557    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-multus-socket-dir-parent\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.374605 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.374581    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-var-lib-openvswitch\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.374605 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.374602    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2-etc-selinux\") pod \"aws-ebs-csi-driver-node-67gtm\" (UID: \"dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67gtm"
Jul 02 13:14:51.375068 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.374630    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4c5abfcf-f610-4f74-9222-a321d08c0a1b-agent-certs\") pod \"konnectivity-agent-7jfsq\" (UID: \"4c5abfcf-f610-4f74-9222-a321d08c0a1b\") " pod="kube-system/konnectivity-agent-7jfsq"
Jul 02 13:14:51.375068 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.374654    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-hostroot\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.375068 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.374680    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/249b4a4b-a5eb-4bc1-8311-1c330010730d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fjbsm\" (UID: \"249b4a4b-a5eb-4bc1-8311-1c330010730d\") " pod="openshift-multus/multus-additional-cni-plugins-fjbsm"
Jul 02 13:14:51.375068 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.374708    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.375068 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.374729    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-cnibin\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.375068 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.374771    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-host-run-k8s-cni-cncf-io\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.375068 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.374732    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-host-run-k8s-cni-cncf-io\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.375068 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.374784    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2-sys-fs\") pod \"aws-ebs-csi-driver-node-67gtm\" (UID: \"dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67gtm"
Jul 02 13:14:51.375068 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.374809    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-host-var-lib-cni-bin\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.375068 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.374836    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-multus-socket-dir-parent\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.375068 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.374873    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-host-var-lib-cni-bin\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.375068 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.374909    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-hostroot\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.375068 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.374944    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-var-lib-openvswitch\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.375068 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375036    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2-etc-selinux\") pod \"aws-ebs-csi-driver-node-67gtm\" (UID: \"dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67gtm"
Jul 02 13:14:51.375764 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375092    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.375764 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.374834    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/249b4a4b-a5eb-4bc1-8311-1c330010730d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fjbsm\" (UID: \"249b4a4b-a5eb-4bc1-8311-1c330010730d\") " pod="openshift-multus/multus-additional-cni-plugins-fjbsm"
Jul 02 13:14:51.375764 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375207    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-host-run-ovn-kubernetes\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.375764 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375238    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-system-cni-dir\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.375764 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375261    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-cni-binary-copy\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.375764 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375282    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/249b4a4b-a5eb-4bc1-8311-1c330010730d-cni-binary-copy\") pod \"multus-additional-cni-plugins-fjbsm\" (UID: \"249b4a4b-a5eb-4bc1-8311-1c330010730d\") " pod="openshift-multus/multus-additional-cni-plugins-fjbsm"
Jul 02 13:14:51.375764 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375284    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-host-var-lib-kubelet\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.375764 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375320    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-host-var-lib-kubelet\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.375764 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375325    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-systemd-units\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.375764 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375338    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-system-cni-dir\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.375764 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375351    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-host-run-netns\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.375764 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375341    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/249b4a4b-a5eb-4bc1-8311-1c330010730d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fjbsm\" (UID: \"249b4a4b-a5eb-4bc1-8311-1c330010730d\") " pod="openshift-multus/multus-additional-cni-plugins-fjbsm"
Jul 02 13:14:51.375764 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375409    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-host-run-ovn-kubernetes\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.375764 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375425    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-host-run-netns\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.375764 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375425    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-os-release\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.375764 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375467    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/249b4a4b-a5eb-4bc1-8311-1c330010730d-os-release\") pod \"multus-additional-cni-plugins-fjbsm\" (UID: \"249b4a4b-a5eb-4bc1-8311-1c330010730d\") " pod="openshift-multus/multus-additional-cni-plugins-fjbsm"
Jul 02 13:14:51.375764 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375485    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-os-release\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.376559 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375514    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-host-var-lib-cni-multus\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.376559 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375531    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/249b4a4b-a5eb-4bc1-8311-1c330010730d-os-release\") pod \"multus-additional-cni-plugins-fjbsm\" (UID: \"249b4a4b-a5eb-4bc1-8311-1c330010730d\") " pod="openshift-multus/multus-additional-cni-plugins-fjbsm"
Jul 02 13:14:51.376559 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375539    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-multus-conf-dir\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.376559 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375424    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-systemd-units\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.376559 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375556    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/249b4a4b-a5eb-4bc1-8311-1c330010730d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fjbsm\" (UID: \"249b4a4b-a5eb-4bc1-8311-1c330010730d\") " pod="openshift-multus/multus-additional-cni-plugins-fjbsm"
Jul 02 13:14:51.376559 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375563    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncsvd\" (UniqueName: \"kubernetes.io/projected/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-kube-api-access-ncsvd\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.376559 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375586    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/249b4a4b-a5eb-4bc1-8311-1c330010730d-system-cni-dir\") pod \"multus-additional-cni-plugins-fjbsm\" (UID: \"249b4a4b-a5eb-4bc1-8311-1c330010730d\") " pod="openshift-multus/multus-additional-cni-plugins-fjbsm"
Jul 02 13:14:51.376559 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375597    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-host-var-lib-cni-multus\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.376559 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375638    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-multus-conf-dir\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.376559 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375636    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/249b4a4b-a5eb-4bc1-8311-1c330010730d-system-cni-dir\") pod \"multus-additional-cni-plugins-fjbsm\" (UID: \"249b4a4b-a5eb-4bc1-8311-1c330010730d\") " pod="openshift-multus/multus-additional-cni-plugins-fjbsm"
Jul 02 13:14:51.376559 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375672    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/249b4a4b-a5eb-4bc1-8311-1c330010730d-cnibin\") pod \"multus-additional-cni-plugins-fjbsm\" (UID: \"249b4a4b-a5eb-4bc1-8311-1c330010730d\") " pod="openshift-multus/multus-additional-cni-plugins-fjbsm"
Jul 02 13:14:51.376559 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375798    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/249b4a4b-a5eb-4bc1-8311-1c330010730d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fjbsm\" (UID: \"249b4a4b-a5eb-4bc1-8311-1c330010730d\") " pod="openshift-multus/multus-additional-cni-plugins-fjbsm"
Jul 02 13:14:51.376559 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375834    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-multus-daemon-config\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.376559 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375858    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-run-ovn\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.376559 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375864    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-cni-binary-copy\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.376559 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375881    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-node-log\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.376559 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375806    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/249b4a4b-a5eb-4bc1-8311-1c330010730d-cnibin\") pod \"multus-additional-cni-plugins-fjbsm\" (UID: \"249b4a4b-a5eb-4bc1-8311-1c330010730d\") " pod="openshift-multus/multus-additional-cni-plugins-fjbsm"
Jul 02 13:14:51.377358 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375904    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-host-cni-bin\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.377358 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375929    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-run-systemd\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.377358 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375937    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-run-ovn\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.377358 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375951    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-log-socket\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.377358 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375957    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/249b4a4b-a5eb-4bc1-8311-1c330010730d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fjbsm\" (UID: \"249b4a4b-a5eb-4bc1-8311-1c330010730d\") " pod="openshift-multus/multus-additional-cni-plugins-fjbsm"
Jul 02 13:14:51.377358 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375972    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-host-cni-bin\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.377358 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.375975    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f0256660-1798-4dba-a311-3705cb73c9ca-env-overrides\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.377358 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376009    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch7rl\" (UniqueName: \"kubernetes.io/projected/f0256660-1798-4dba-a311-3705cb73c9ca-kube-api-access-ch7rl\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.377358 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376010    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-node-log\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.377358 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376035    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2-registration-dir\") pod \"aws-ebs-csi-driver-node-67gtm\" (UID: \"dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67gtm"
Jul 02 13:14:51.377358 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376042    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-log-socket\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.377358 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376011    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-run-systemd\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.377358 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376060    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4c5abfcf-f610-4f74-9222-a321d08c0a1b-konnectivity-ca\") pod \"konnectivity-agent-7jfsq\" (UID: \"4c5abfcf-f610-4f74-9222-a321d08c0a1b\") " pod="kube-system/konnectivity-agent-7jfsq"
Jul 02 13:14:51.377358 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376085    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-etc-kubernetes\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.377358 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376102    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2-registration-dir\") pod \"aws-ebs-csi-driver-node-67gtm\" (UID: \"dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67gtm"
Jul 02 13:14:51.377358 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376108    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-host-kubelet\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.377358 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376134    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-host-slash\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.378146 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376158    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f0256660-1798-4dba-a311-3705cb73c9ca-ovnkube-script-lib\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.378146 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376185    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-host-run-multus-certs\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.378146 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376223    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-run-openvswitch\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.378146 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376246    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-etc-kubernetes\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.378146 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376261    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-host-cni-netd\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.378146 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376288    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-multus-daemon-config\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.378146 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376319    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-host-slash\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.378146 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376284    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f0256660-1798-4dba-a311-3705cb73c9ca-ovnkube-config\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.378146 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376334    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-host-cni-netd\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.378146 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376337    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-run-openvswitch\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.378146 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376290    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-host-run-multus-certs\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.378146 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376363    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-multus-cni-dir\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.378146 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376408    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f0256660-1798-4dba-a311-3705cb73c9ca-ovn-node-metrics-cert\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.378146 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376426    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0256660-1798-4dba-a311-3705cb73c9ca-host-kubelet\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.378146 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376426    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f0256660-1798-4dba-a311-3705cb73c9ca-env-overrides\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.378146 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376487    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-67gtm\" (UID: \"dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67gtm"
Jul 02 13:14:51.378146 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376541    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-multus-cni-dir\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.378146 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376433    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-67gtm\" (UID: \"dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67gtm"
Jul 02 13:14:51.378912 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376580    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2-socket-dir\") pod \"aws-ebs-csi-driver-node-67gtm\" (UID: \"dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67gtm"
Jul 02 13:14:51.378912 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376605    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2-device-dir\") pod \"aws-ebs-csi-driver-node-67gtm\" (UID: \"dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67gtm"
Jul 02 13:14:51.378912 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376647    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2r82\" (UniqueName: \"kubernetes.io/projected/159f78cc-1fe0-40ca-a0cd-92254294389c-kube-api-access-s2r82\") pod \"network-check-target-tx24z\" (UID: \"159f78cc-1fe0-40ca-a0cd-92254294389c\") " pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:14:51.378912 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376654    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4c5abfcf-f610-4f74-9222-a321d08c0a1b-konnectivity-ca\") pod \"konnectivity-agent-7jfsq\" (UID: \"4c5abfcf-f610-4f74-9222-a321d08c0a1b\") " pod="kube-system/konnectivity-agent-7jfsq"
Jul 02 13:14:51.378912 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376753    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2-device-dir\") pod \"aws-ebs-csi-driver-node-67gtm\" (UID: \"dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67gtm"
Jul 02 13:14:51.378912 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376759    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2-socket-dir\") pod \"aws-ebs-csi-driver-node-67gtm\" (UID: \"dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67gtm"
Jul 02 13:14:51.378912 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376801    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f0256660-1798-4dba-a311-3705cb73c9ca-ovnkube-config\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.378912 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.376834    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f0256660-1798-4dba-a311-3705cb73c9ca-ovnkube-script-lib\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.378912 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.378527    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4c5abfcf-f610-4f74-9222-a321d08c0a1b-agent-certs\") pod \"konnectivity-agent-7jfsq\" (UID: \"4c5abfcf-f610-4f74-9222-a321d08c0a1b\") " pod="kube-system/konnectivity-agent-7jfsq"
Jul 02 13:14:51.378912 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.378912    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f0256660-1798-4dba-a311-3705cb73c9ca-ovn-node-metrics-cert\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.393498 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:51.393407    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:14:51.393498 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:51.393440    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:14:51.393498 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:51.393462    2583 projected.go:196] Error preparing data for projected volume kube-api-access-s2r82 for pod openshift-network-diagnostics/network-check-target-tx24z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:51.393727 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:51.393552    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/159f78cc-1fe0-40ca-a0cd-92254294389c-kube-api-access-s2r82 podName:159f78cc-1fe0-40ca-a0cd-92254294389c nodeName:}" failed. No retries permitted until 2026-07-02 13:14:51.893531442 +0000 UTC m=+3.121442600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-s2r82" (UniqueName: "kubernetes.io/projected/159f78cc-1fe0-40ca-a0cd-92254294389c-kube-api-access-s2r82") pod "network-check-target-tx24z" (UID: "159f78cc-1fe0-40ca-a0cd-92254294389c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:51.398845 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.398769    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch7rl\" (UniqueName: \"kubernetes.io/projected/f0256660-1798-4dba-a311-3705cb73c9ca-kube-api-access-ch7rl\") pod \"ovnkube-node-jlkds\" (UID: \"f0256660-1798-4dba-a311-3705cb73c9ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.398845 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.398769    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8qrj\" (UniqueName: \"kubernetes.io/projected/249b4a4b-a5eb-4bc1-8311-1c330010730d-kube-api-access-b8qrj\") pod \"multus-additional-cni-plugins-fjbsm\" (UID: \"249b4a4b-a5eb-4bc1-8311-1c330010730d\") " pod="openshift-multus/multus-additional-cni-plugins-fjbsm"
Jul 02 13:14:51.399754 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.399727    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncsvd\" (UniqueName: \"kubernetes.io/projected/d326b4cb-cbbe-41fc-8de2-e1876c60c9df-kube-api-access-ncsvd\") pod \"multus-2hwvd\" (UID: \"d326b4cb-cbbe-41fc-8de2-e1876c60c9df\") " pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.400755 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.400722    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fl8g\" (UniqueName: \"kubernetes.io/projected/dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2-kube-api-access-6fl8g\") pod \"aws-ebs-csi-driver-node-67gtm\" (UID: \"dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67gtm"
Jul 02 13:14:51.467110 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.467079    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9m4tg"
Jul 02 13:14:51.475958 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.475931    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6rfcq"
Jul 02 13:14:51.485894 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.485867    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hzspc"
Jul 02 13:14:51.491771 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.491747    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9pvbh"
Jul 02 13:14:51.498170 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.498146    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fjbsm"
Jul 02 13:14:51.505986 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.505955    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7jfsq"
Jul 02 13:14:51.514246 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.514217    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:14:51.522190 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.522162    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67gtm"
Jul 02 13:14:51.528164 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.528139    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2hwvd"
Jul 02 13:14:51.778955 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.778880    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bfa11a9f-ca92-4214-8e84-0b9c48a8337c-original-pull-secret\") pod \"global-pull-secret-syncer-9fpt5\" (UID: \"bfa11a9f-ca92-4214-8e84-0b9c48a8337c\") " pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:14:51.779113 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.778953    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cae18b4-8f5b-4ff7-a73f-fc668af0a663-metrics-certs\") pod \"network-metrics-daemon-xzl6q\" (UID: \"0cae18b4-8f5b-4ff7-a73f-fc668af0a663\") " pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:14:51.779113 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:51.779048    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:51.779113 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:51.779076    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:51.779113 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:51.779107    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfa11a9f-ca92-4214-8e84-0b9c48a8337c-original-pull-secret podName:bfa11a9f-ca92-4214-8e84-0b9c48a8337c nodeName:}" failed. No retries permitted until 2026-07-02 13:14:52.779092026 +0000 UTC m=+4.007003178 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bfa11a9f-ca92-4214-8e84-0b9c48a8337c-original-pull-secret") pod "global-pull-secret-syncer-9fpt5" (UID: "bfa11a9f-ca92-4214-8e84-0b9c48a8337c") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:51.779269 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:51.779123    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cae18b4-8f5b-4ff7-a73f-fc668af0a663-metrics-certs podName:0cae18b4-8f5b-4ff7-a73f-fc668af0a663 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:52.779116219 +0000 UTC m=+4.007027370 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cae18b4-8f5b-4ff7-a73f-fc668af0a663-metrics-certs") pod "network-metrics-daemon-xzl6q" (UID: "0cae18b4-8f5b-4ff7-a73f-fc668af0a663") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:51.979946 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:51.979905    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2r82\" (UniqueName: \"kubernetes.io/projected/159f78cc-1fe0-40ca-a0cd-92254294389c-kube-api-access-s2r82\") pod \"network-check-target-tx24z\" (UID: \"159f78cc-1fe0-40ca-a0cd-92254294389c\") " pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:14:51.980121 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:51.980075    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:14:51.980121 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:51.980098    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:14:51.980121 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:51.980111    2583 projected.go:196] Error preparing data for projected volume kube-api-access-s2r82 for pod openshift-network-diagnostics/network-check-target-tx24z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:51.980249 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:51.980180    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/159f78cc-1fe0-40ca-a0cd-92254294389c-kube-api-access-s2r82 podName:159f78cc-1fe0-40ca-a0cd-92254294389c nodeName:}" failed. No retries permitted until 2026-07-02 13:14:52.980160465 +0000 UTC m=+4.208071617 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-s2r82" (UniqueName: "kubernetes.io/projected/159f78cc-1fe0-40ca-a0cd-92254294389c-kube-api-access-s2r82") pod "network-check-target-tx24z" (UID: "159f78cc-1fe0-40ca-a0cd-92254294389c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:52.154976 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:52.154822    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod249b4a4b_a5eb_4bc1_8311_1c330010730d.slice/crio-099310234bfc286d8d4c823f39c33e056f566ada06296b10fb05aa46a8cc867c WatchSource:0}: Error finding container 099310234bfc286d8d4c823f39c33e056f566ada06296b10fb05aa46a8cc867c: Status 404 returned error can't find the container with id 099310234bfc286d8d4c823f39c33e056f566ada06296b10fb05aa46a8cc867c
Jul 02 13:14:52.157165 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:52.156977    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc9c602d_4c56_4fe4_b1d7_5ef367ca6382.slice/crio-593d45125b665bdf59a4d8bbab554344e8a84af66988b262121941f6e4589fe5 WatchSource:0}: Error finding container 593d45125b665bdf59a4d8bbab554344e8a84af66988b262121941f6e4589fe5: Status 404 returned error can't find the container with id 593d45125b665bdf59a4d8bbab554344e8a84af66988b262121941f6e4589fe5
Jul 02 13:14:52.160153 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:52.160125    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9be48a_a6d9_4ea4_b664_b7c2ffe980b1.slice/crio-9bad1f473a688b1cc145fbd7909bdacedc351011cf85c6b54adff7c343625758 WatchSource:0}: Error finding container 9bad1f473a688b1cc145fbd7909bdacedc351011cf85c6b54adff7c343625758: Status 404 returned error can't find the container with id 9bad1f473a688b1cc145fbd7909bdacedc351011cf85c6b54adff7c343625758
Jul 02 13:14:52.161615 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:52.161587    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda7049b_02ce_4b45_8ffe_18a5ffe9b6c2.slice/crio-1fbcc7839042f7931e2b7595824fbcb67df24404777d9900f431023382433451 WatchSource:0}: Error finding container 1fbcc7839042f7931e2b7595824fbcb67df24404777d9900f431023382433451: Status 404 returned error can't find the container with id 1fbcc7839042f7931e2b7595824fbcb67df24404777d9900f431023382433451
Jul 02 13:14:52.162483 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:52.162371    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2cccfac_f483_4a64_8788_b78393b5a88e.slice/crio-7ebe766b7757607a698da53f661ba03bc5455960a68b18f169dfc47e5913f613 WatchSource:0}: Error finding container 7ebe766b7757607a698da53f661ba03bc5455960a68b18f169dfc47e5913f613: Status 404 returned error can't find the container with id 7ebe766b7757607a698da53f661ba03bc5455960a68b18f169dfc47e5913f613
Jul 02 13:14:52.164075 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:52.163234    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2b80dbb_5bb9_4d5e_add9_aeb0519f92c1.slice/crio-422b9a443628f1e99e6c15d46987bb4d0356adab3e6c25e4f34ebc437932b086 WatchSource:0}: Error finding container 422b9a443628f1e99e6c15d46987bb4d0356adab3e6c25e4f34ebc437932b086: Status 404 returned error can't find the container with id 422b9a443628f1e99e6c15d46987bb4d0356adab3e6c25e4f34ebc437932b086
Jul 02 13:14:52.166750 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:14:52.166595    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0256660_1798_4dba_a311_3705cb73c9ca.slice/crio-18366af2c0c9c628b2b7333ce8cb93693134a596994398b141f6049699b6a50a WatchSource:0}: Error finding container 18366af2c0c9c628b2b7333ce8cb93693134a596994398b141f6049699b6a50a: Status 404 returned error can't find the container with id 18366af2c0c9c628b2b7333ce8cb93693134a596994398b141f6049699b6a50a
Jul 02 13:14:52.219765 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:52.219718    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 13:09:51 +0000 UTC" deadline="2028-03-03 13:34:11.504528591 +0000 UTC"
Jul 02 13:14:52.219765 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:52.219759    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14640h19m19.284771198s"
Jul 02 13:14:52.236981 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:52.236730    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jlkds" event={"ID":"f0256660-1798-4dba-a311-3705cb73c9ca","Type":"ContainerStarted","Data":"18366af2c0c9c628b2b7333ce8cb93693134a596994398b141f6049699b6a50a"}
Jul 02 13:14:52.238001 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:52.237971    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7jfsq" event={"ID":"4c5abfcf-f610-4f74-9222-a321d08c0a1b","Type":"ContainerStarted","Data":"ca13cda20aa43910e375e0751f7e13907161a395e1569e83efeef4931215cd77"}
Jul 02 13:14:52.239410 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:52.239370    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67gtm" event={"ID":"dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2","Type":"ContainerStarted","Data":"1fbcc7839042f7931e2b7595824fbcb67df24404777d9900f431023382433451"}
Jul 02 13:14:52.241334 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:52.241279    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6rfcq" event={"ID":"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1","Type":"ContainerStarted","Data":"9bad1f473a688b1cc145fbd7909bdacedc351011cf85c6b54adff7c343625758"}
Jul 02 13:14:52.243038 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:52.243014    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2hwvd" event={"ID":"d326b4cb-cbbe-41fc-8de2-e1876c60c9df","Type":"ContainerStarted","Data":"8dc325fc9db2bfc91f8adb8c5333d0240d11e03a22b96c578ce0288496648ac4"}
Jul 02 13:14:52.245059 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:52.244881    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hzspc" event={"ID":"a2b80dbb-5bb9-4d5e-add9-aeb0519f92c1","Type":"ContainerStarted","Data":"422b9a443628f1e99e6c15d46987bb4d0356adab3e6c25e4f34ebc437932b086"}
Jul 02 13:14:52.246067 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:52.246041    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9pvbh" event={"ID":"b2cccfac-f483-4a64-8788-b78393b5a88e","Type":"ContainerStarted","Data":"7ebe766b7757607a698da53f661ba03bc5455960a68b18f169dfc47e5913f613"}
Jul 02 13:14:52.247205 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:52.247178    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9m4tg" event={"ID":"dc9c602d-4c56-4fe4-b1d7-5ef367ca6382","Type":"ContainerStarted","Data":"593d45125b665bdf59a4d8bbab554344e8a84af66988b262121941f6e4589fe5"}
Jul 02 13:14:52.248610 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:52.248587    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjbsm" event={"ID":"249b4a4b-a5eb-4bc1-8311-1c330010730d","Type":"ContainerStarted","Data":"099310234bfc286d8d4c823f39c33e056f566ada06296b10fb05aa46a8cc867c"}
Jul 02 13:14:52.786190 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:52.786103    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cae18b4-8f5b-4ff7-a73f-fc668af0a663-metrics-certs\") pod \"network-metrics-daemon-xzl6q\" (UID: \"0cae18b4-8f5b-4ff7-a73f-fc668af0a663\") " pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:14:52.786357 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:52.786188    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bfa11a9f-ca92-4214-8e84-0b9c48a8337c-original-pull-secret\") pod \"global-pull-secret-syncer-9fpt5\" (UID: \"bfa11a9f-ca92-4214-8e84-0b9c48a8337c\") " pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:14:52.786357 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:52.786303    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:52.786639 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:52.786362    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfa11a9f-ca92-4214-8e84-0b9c48a8337c-original-pull-secret podName:bfa11a9f-ca92-4214-8e84-0b9c48a8337c nodeName:}" failed. No retries permitted until 2026-07-02 13:14:54.786345186 +0000 UTC m=+6.014256338 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bfa11a9f-ca92-4214-8e84-0b9c48a8337c-original-pull-secret") pod "global-pull-secret-syncer-9fpt5" (UID: "bfa11a9f-ca92-4214-8e84-0b9c48a8337c") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:52.786639 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:52.786446    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:52.786639 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:52.786485    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cae18b4-8f5b-4ff7-a73f-fc668af0a663-metrics-certs podName:0cae18b4-8f5b-4ff7-a73f-fc668af0a663 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:54.786472898 +0000 UTC m=+6.014384053 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cae18b4-8f5b-4ff7-a73f-fc668af0a663-metrics-certs") pod "network-metrics-daemon-xzl6q" (UID: "0cae18b4-8f5b-4ff7-a73f-fc668af0a663") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:52.988188 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:52.988149    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2r82\" (UniqueName: \"kubernetes.io/projected/159f78cc-1fe0-40ca-a0cd-92254294389c-kube-api-access-s2r82\") pod \"network-check-target-tx24z\" (UID: \"159f78cc-1fe0-40ca-a0cd-92254294389c\") " pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:14:52.988369 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:52.988330    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:14:52.988369 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:52.988350    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:14:52.988369 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:52.988361    2583 projected.go:196] Error preparing data for projected volume kube-api-access-s2r82 for pod openshift-network-diagnostics/network-check-target-tx24z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:52.988584 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:52.988439    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/159f78cc-1fe0-40ca-a0cd-92254294389c-kube-api-access-s2r82 podName:159f78cc-1fe0-40ca-a0cd-92254294389c nodeName:}" failed. No retries permitted until 2026-07-02 13:14:54.988419562 +0000 UTC m=+6.216330719 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-s2r82" (UniqueName: "kubernetes.io/projected/159f78cc-1fe0-40ca-a0cd-92254294389c-kube-api-access-s2r82") pod "network-check-target-tx24z" (UID: "159f78cc-1fe0-40ca-a0cd-92254294389c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:53.220569 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:53.220480    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 13:09:51 +0000 UTC" deadline="2028-03-08 03:37:34.420276828 +0000 UTC"
Jul 02 13:14:53.220569 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:53.220521    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14750h22m41.199761053s"
Jul 02 13:14:53.229065 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:53.228516    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:14:53.229065 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:53.228526    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:14:53.229065 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:53.228668    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzl6q" podUID="0cae18b4-8f5b-4ff7-a73f-fc668af0a663"
Jul 02 13:14:53.229065 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:53.228727    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fpt5" podUID="bfa11a9f-ca92-4214-8e84-0b9c48a8337c"
Jul 02 13:14:53.229065 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:53.228811    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:14:53.229065 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:53.228965    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx24z" podUID="159f78cc-1fe0-40ca-a0cd-92254294389c"
Jul 02 13:14:53.278710 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:53.278675    2583 generic.go:356] "Generic (PLEG): container finished" podID="ef0c8167d6f730479020bb6b4e702a8c" containerID="54daf00934c0936b50f5abeb08b7b307aec0fccc07f9e2c410d5856b154ff06c" exitCode=0
Jul 02 13:14:53.279045 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:53.279009    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal" event={"ID":"ef0c8167d6f730479020bb6b4e702a8c","Type":"ContainerDied","Data":"54daf00934c0936b50f5abeb08b7b307aec0fccc07f9e2c410d5856b154ff06c"}
Jul 02 13:14:53.291277 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:53.291234    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal" event={"ID":"b72efbb5a6c47d6a170c394c0a860619","Type":"ContainerStarted","Data":"4a63ea0d16515eba2ce6dc8799ace85e26d0546559acbf7db2b2d4a3ac55a1bd"}
Jul 02 13:14:54.307050 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:54.307009    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal" event={"ID":"ef0c8167d6f730479020bb6b4e702a8c","Type":"ContainerStarted","Data":"de963211ac8f116e92b875b5357189f3cd71265ca91fc10e6484af73bb28acad"}
Jul 02 13:14:54.328557 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:54.328192    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-143.ec2.internal" podStartSLOduration=4.328152005 podStartE2EDuration="4.328152005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:14:53.318582048 +0000 UTC m=+4.546493216" watchObservedRunningTime="2026-07-02 13:14:54.328152005 +0000 UTC m=+5.556063181"
Jul 02 13:14:54.801901 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:54.801861    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cae18b4-8f5b-4ff7-a73f-fc668af0a663-metrics-certs\") pod \"network-metrics-daemon-xzl6q\" (UID: \"0cae18b4-8f5b-4ff7-a73f-fc668af0a663\") " pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:14:54.802067 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:54.801947    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bfa11a9f-ca92-4214-8e84-0b9c48a8337c-original-pull-secret\") pod \"global-pull-secret-syncer-9fpt5\" (UID: \"bfa11a9f-ca92-4214-8e84-0b9c48a8337c\") " pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:14:54.802151 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:54.802096    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:54.802206 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:54.802175    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfa11a9f-ca92-4214-8e84-0b9c48a8337c-original-pull-secret podName:bfa11a9f-ca92-4214-8e84-0b9c48a8337c nodeName:}" failed. No retries permitted until 2026-07-02 13:14:58.802157276 +0000 UTC m=+10.030068431 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bfa11a9f-ca92-4214-8e84-0b9c48a8337c-original-pull-secret") pod "global-pull-secret-syncer-9fpt5" (UID: "bfa11a9f-ca92-4214-8e84-0b9c48a8337c") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:54.802603 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:54.802580    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:54.802689 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:54.802639    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cae18b4-8f5b-4ff7-a73f-fc668af0a663-metrics-certs podName:0cae18b4-8f5b-4ff7-a73f-fc668af0a663 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:58.80262338 +0000 UTC m=+10.030534535 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cae18b4-8f5b-4ff7-a73f-fc668af0a663-metrics-certs") pod "network-metrics-daemon-xzl6q" (UID: "0cae18b4-8f5b-4ff7-a73f-fc668af0a663") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:55.003635 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:55.003594    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2r82\" (UniqueName: \"kubernetes.io/projected/159f78cc-1fe0-40ca-a0cd-92254294389c-kube-api-access-s2r82\") pod \"network-check-target-tx24z\" (UID: \"159f78cc-1fe0-40ca-a0cd-92254294389c\") " pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:14:55.003813 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:55.003802    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:14:55.003874 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:55.003821    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:14:55.003874 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:55.003833    2583 projected.go:196] Error preparing data for projected volume kube-api-access-s2r82 for pod openshift-network-diagnostics/network-check-target-tx24z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:55.003972 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:55.003892    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/159f78cc-1fe0-40ca-a0cd-92254294389c-kube-api-access-s2r82 podName:159f78cc-1fe0-40ca-a0cd-92254294389c nodeName:}" failed. No retries permitted until 2026-07-02 13:14:59.003875146 +0000 UTC m=+10.231786313 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-s2r82" (UniqueName: "kubernetes.io/projected/159f78cc-1fe0-40ca-a0cd-92254294389c-kube-api-access-s2r82") pod "network-check-target-tx24z" (UID: "159f78cc-1fe0-40ca-a0cd-92254294389c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:55.228775 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:55.228402    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:14:55.228775 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:55.228411    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:14:55.228775 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:55.228513    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzl6q" podUID="0cae18b4-8f5b-4ff7-a73f-fc668af0a663"
Jul 02 13:14:55.228775 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:55.228604    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fpt5" podUID="bfa11a9f-ca92-4214-8e84-0b9c48a8337c"
Jul 02 13:14:55.228775 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:55.228668    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:14:55.228775 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:55.228732    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx24z" podUID="159f78cc-1fe0-40ca-a0cd-92254294389c"
Jul 02 13:14:57.229130 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:57.229097    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:14:57.229623 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:57.229105    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:14:57.229623 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:57.229225    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx24z" podUID="159f78cc-1fe0-40ca-a0cd-92254294389c"
Jul 02 13:14:57.229623 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:57.229303    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzl6q" podUID="0cae18b4-8f5b-4ff7-a73f-fc668af0a663"
Jul 02 13:14:57.229623 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:57.229323    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:14:57.229623 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:57.229495    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fpt5" podUID="bfa11a9f-ca92-4214-8e84-0b9c48a8337c"
Jul 02 13:14:58.838323 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:58.838281    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cae18b4-8f5b-4ff7-a73f-fc668af0a663-metrics-certs\") pod \"network-metrics-daemon-xzl6q\" (UID: \"0cae18b4-8f5b-4ff7-a73f-fc668af0a663\") " pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:14:58.838842 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:58.838373    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bfa11a9f-ca92-4214-8e84-0b9c48a8337c-original-pull-secret\") pod \"global-pull-secret-syncer-9fpt5\" (UID: \"bfa11a9f-ca92-4214-8e84-0b9c48a8337c\") " pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:14:58.838842 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:58.838444    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:58.838842 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:58.838516    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:58.838842 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:58.838531    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cae18b4-8f5b-4ff7-a73f-fc668af0a663-metrics-certs podName:0cae18b4-8f5b-4ff7-a73f-fc668af0a663 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:06.83850898 +0000 UTC m=+18.066420147 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cae18b4-8f5b-4ff7-a73f-fc668af0a663-metrics-certs") pod "network-metrics-daemon-xzl6q" (UID: "0cae18b4-8f5b-4ff7-a73f-fc668af0a663") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:58.838842 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:58.838630    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfa11a9f-ca92-4214-8e84-0b9c48a8337c-original-pull-secret podName:bfa11a9f-ca92-4214-8e84-0b9c48a8337c nodeName:}" failed. No retries permitted until 2026-07-02 13:15:06.83861251 +0000 UTC m=+18.066523664 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bfa11a9f-ca92-4214-8e84-0b9c48a8337c-original-pull-secret") pod "global-pull-secret-syncer-9fpt5" (UID: "bfa11a9f-ca92-4214-8e84-0b9c48a8337c") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:59.039809 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:59.039752    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2r82\" (UniqueName: \"kubernetes.io/projected/159f78cc-1fe0-40ca-a0cd-92254294389c-kube-api-access-s2r82\") pod \"network-check-target-tx24z\" (UID: \"159f78cc-1fe0-40ca-a0cd-92254294389c\") " pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:14:59.039969 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:59.039914    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:14:59.039969 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:59.039937    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:14:59.039969 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:59.039951    2583 projected.go:196] Error preparing data for projected volume kube-api-access-s2r82 for pod openshift-network-diagnostics/network-check-target-tx24z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:59.040162 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:59.040005    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/159f78cc-1fe0-40ca-a0cd-92254294389c-kube-api-access-s2r82 podName:159f78cc-1fe0-40ca-a0cd-92254294389c nodeName:}" failed. No retries permitted until 2026-07-02 13:15:07.039991179 +0000 UTC m=+18.267902331 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-s2r82" (UniqueName: "kubernetes.io/projected/159f78cc-1fe0-40ca-a0cd-92254294389c-kube-api-access-s2r82") pod "network-check-target-tx24z" (UID: "159f78cc-1fe0-40ca-a0cd-92254294389c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:59.227751 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:59.227725    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:14:59.227962 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:59.227874    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:14:59.229125 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:59.229046    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx24z" podUID="159f78cc-1fe0-40ca-a0cd-92254294389c"
Jul 02 13:14:59.229470 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:59.229448    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzl6q" podUID="0cae18b4-8f5b-4ff7-a73f-fc668af0a663"
Jul 02 13:14:59.229562 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:14:59.229510    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:14:59.229628 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:14:59.229584    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fpt5" podUID="bfa11a9f-ca92-4214-8e84-0b9c48a8337c"
Jul 02 13:15:01.227623 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:01.227583    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:15:01.228082 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:01.227715    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:15:01.228082 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:01.227600    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:15:01.228082 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:01.227831    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fpt5" podUID="bfa11a9f-ca92-4214-8e84-0b9c48a8337c"
Jul 02 13:15:01.228082 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:01.227711    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx24z" podUID="159f78cc-1fe0-40ca-a0cd-92254294389c"
Jul 02 13:15:01.228082 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:01.227992    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzl6q" podUID="0cae18b4-8f5b-4ff7-a73f-fc668af0a663"
Jul 02 13:15:03.227725 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:03.227692    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:15:03.228207 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:03.227692    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:15:03.228207 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:03.227822    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fpt5" podUID="bfa11a9f-ca92-4214-8e84-0b9c48a8337c"
Jul 02 13:15:03.228207 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:03.227692    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:15:03.228207 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:03.227879    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzl6q" podUID="0cae18b4-8f5b-4ff7-a73f-fc668af0a663"
Jul 02 13:15:03.228207 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:03.227974    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx24z" podUID="159f78cc-1fe0-40ca-a0cd-92254294389c"
Jul 02 13:15:05.227905 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:05.227873    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:15:05.228412 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:05.227874    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:15:05.228412 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:05.228005    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzl6q" podUID="0cae18b4-8f5b-4ff7-a73f-fc668af0a663"
Jul 02 13:15:05.228412 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:05.227874    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:15:05.228412 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:05.228083    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx24z" podUID="159f78cc-1fe0-40ca-a0cd-92254294389c"
Jul 02 13:15:05.228412 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:05.228160    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fpt5" podUID="bfa11a9f-ca92-4214-8e84-0b9c48a8337c"
Jul 02 13:15:06.898955 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:06.898909    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cae18b4-8f5b-4ff7-a73f-fc668af0a663-metrics-certs\") pod \"network-metrics-daemon-xzl6q\" (UID: \"0cae18b4-8f5b-4ff7-a73f-fc668af0a663\") " pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:15:06.899492 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:06.899017    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bfa11a9f-ca92-4214-8e84-0b9c48a8337c-original-pull-secret\") pod \"global-pull-secret-syncer-9fpt5\" (UID: \"bfa11a9f-ca92-4214-8e84-0b9c48a8337c\") " pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:15:06.899492 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:06.899104    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:15:06.899492 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:06.899134    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:15:06.899492 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:06.899185    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cae18b4-8f5b-4ff7-a73f-fc668af0a663-metrics-certs podName:0cae18b4-8f5b-4ff7-a73f-fc668af0a663 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:22.89916349 +0000 UTC m=+34.127074664 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cae18b4-8f5b-4ff7-a73f-fc668af0a663-metrics-certs") pod "network-metrics-daemon-xzl6q" (UID: "0cae18b4-8f5b-4ff7-a73f-fc668af0a663") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:15:06.899492 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:06.899205    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfa11a9f-ca92-4214-8e84-0b9c48a8337c-original-pull-secret podName:bfa11a9f-ca92-4214-8e84-0b9c48a8337c nodeName:}" failed. No retries permitted until 2026-07-02 13:15:22.899195769 +0000 UTC m=+34.127106921 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bfa11a9f-ca92-4214-8e84-0b9c48a8337c-original-pull-secret") pod "global-pull-secret-syncer-9fpt5" (UID: "bfa11a9f-ca92-4214-8e84-0b9c48a8337c") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:15:07.100857 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:07.100818    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2r82\" (UniqueName: \"kubernetes.io/projected/159f78cc-1fe0-40ca-a0cd-92254294389c-kube-api-access-s2r82\") pod \"network-check-target-tx24z\" (UID: \"159f78cc-1fe0-40ca-a0cd-92254294389c\") " pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:15:07.101024 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:07.100958    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:15:07.101024 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:07.100973    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:15:07.101024 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:07.100982    2583 projected.go:196] Error preparing data for projected volume kube-api-access-s2r82 for pod openshift-network-diagnostics/network-check-target-tx24z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:15:07.101130 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:07.101030    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/159f78cc-1fe0-40ca-a0cd-92254294389c-kube-api-access-s2r82 podName:159f78cc-1fe0-40ca-a0cd-92254294389c nodeName:}" failed. No retries permitted until 2026-07-02 13:15:23.101017954 +0000 UTC m=+34.328929105 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-s2r82" (UniqueName: "kubernetes.io/projected/159f78cc-1fe0-40ca-a0cd-92254294389c-kube-api-access-s2r82") pod "network-check-target-tx24z" (UID: "159f78cc-1fe0-40ca-a0cd-92254294389c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:15:07.228195 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:07.228110    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:15:07.228347 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:07.228110    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:15:07.228347 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:07.228245    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx24z" podUID="159f78cc-1fe0-40ca-a0cd-92254294389c"
Jul 02 13:15:07.228347 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:07.228263    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:15:07.228544 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:07.228366    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzl6q" podUID="0cae18b4-8f5b-4ff7-a73f-fc668af0a663"
Jul 02 13:15:07.228544 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:07.228476    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fpt5" podUID="bfa11a9f-ca92-4214-8e84-0b9c48a8337c"
Jul 02 13:15:09.228403 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:09.228359    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:15:09.228810 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:09.228487    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:15:09.228810 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:09.228507    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:15:09.228810 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:09.228485    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx24z" podUID="159f78cc-1fe0-40ca-a0cd-92254294389c"
Jul 02 13:15:09.228810 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:09.228606    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzl6q" podUID="0cae18b4-8f5b-4ff7-a73f-fc668af0a663"
Jul 02 13:15:09.228810 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:09.228716    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fpt5" podUID="bfa11a9f-ca92-4214-8e84-0b9c48a8337c"
Jul 02 13:15:11.228362 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:11.228177    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:15:11.228983 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:11.228463    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx24z" podUID="159f78cc-1fe0-40ca-a0cd-92254294389c"
Jul 02 13:15:11.228983 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:11.228190    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:15:11.228983 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:11.228177    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:15:11.228983 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:11.228546    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzl6q" podUID="0cae18b4-8f5b-4ff7-a73f-fc668af0a663"
Jul 02 13:15:11.228983 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:11.228592    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fpt5" podUID="bfa11a9f-ca92-4214-8e84-0b9c48a8337c"
Jul 02 13:15:11.345001 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:11.344967    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hzspc" event={"ID":"a2b80dbb-5bb9-4d5e-add9-aeb0519f92c1","Type":"ContainerStarted","Data":"830f383a74f75896ae3c519788611c22e665db85ee5b867dcf453a1e022b62f1"}
Jul 02 13:15:11.346379 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:11.346352    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9pvbh" event={"ID":"b2cccfac-f483-4a64-8788-b78393b5a88e","Type":"ContainerStarted","Data":"a6ad4a3a1081c70e225ba4aea6cee365094e6f71af8fd9818320deb94bac6e0a"}
Jul 02 13:15:11.347806 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:11.347778    2583 generic.go:356] "Generic (PLEG): container finished" podID="249b4a4b-a5eb-4bc1-8311-1c330010730d" containerID="8c4e0274b34a0d4a7ad07e2f12216f7b519302eca7491d6814027fa721ce4055" exitCode=0
Jul 02 13:15:11.347910 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:11.347853    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjbsm" event={"ID":"249b4a4b-a5eb-4bc1-8311-1c330010730d","Type":"ContainerDied","Data":"8c4e0274b34a0d4a7ad07e2f12216f7b519302eca7491d6814027fa721ce4055"}
Jul 02 13:15:11.350381 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:11.350359    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 13:15:11.350791 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:11.350766    2583 generic.go:356] "Generic (PLEG): container finished" podID="f0256660-1798-4dba-a311-3705cb73c9ca" containerID="e2746b9b4e241dc8f1b6788aa49b2dcc5ac29397f36f6da2d22d76d52e12b74a" exitCode=1
Jul 02 13:15:11.350867 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:11.350840    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jlkds" event={"ID":"f0256660-1798-4dba-a311-3705cb73c9ca","Type":"ContainerStarted","Data":"929d0fe006a7903a76959b570af3409eb3cc7960ecf9e9a0184a20b50647f137"}
Jul 02 13:15:11.350933 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:11.350919    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jlkds" event={"ID":"f0256660-1798-4dba-a311-3705cb73c9ca","Type":"ContainerStarted","Data":"687b6d5c0a8468b4ca44d44b19f8115ce6046c23b3eca4474bdb5e071e8c240b"}
Jul 02 13:15:11.350988 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:11.350974    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jlkds" event={"ID":"f0256660-1798-4dba-a311-3705cb73c9ca","Type":"ContainerDied","Data":"e2746b9b4e241dc8f1b6788aa49b2dcc5ac29397f36f6da2d22d76d52e12b74a"}
Jul 02 13:15:11.351055 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:11.351037    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jlkds" event={"ID":"f0256660-1798-4dba-a311-3705cb73c9ca","Type":"ContainerStarted","Data":"b83f4c432c2f801d9c2fe09d2c174c233d5cf7405c74bfc70fddb6c15addf3cf"}
Jul 02 13:15:11.352246 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:11.352216    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7jfsq" event={"ID":"4c5abfcf-f610-4f74-9222-a321d08c0a1b","Type":"ContainerStarted","Data":"3364efaaaf51a9ab19f4d8a04ac31f0b32b2193c51fc6c9d7e33276051c1bce5"}
Jul 02 13:15:11.353654 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:11.353631    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67gtm" event={"ID":"dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2","Type":"ContainerStarted","Data":"6c097a2641d2bf14b84f2e773c617e07fe0427d038e1d232ef1a57f425984a20"}
Jul 02 13:15:11.355064 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:11.355045    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6rfcq" event={"ID":"ce9be48a-a6d9-4ea4-b664-b7c2ffe980b1","Type":"ContainerStarted","Data":"90da3f2dd59e9ada06b5c93c3c966a4c28e47a43e9339393379ff4a8385ef956"}
Jul 02 13:15:11.356517 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:11.356499    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2hwvd" event={"ID":"d326b4cb-cbbe-41fc-8de2-e1876c60c9df","Type":"ContainerStarted","Data":"c9ec974a6cc6f1ec1f672c9b1d9431021beb692de9ede41be0d485ef4895e93c"}
Jul 02 13:15:11.369368 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:11.369316    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-143.ec2.internal" podStartSLOduration=21.369302098 podStartE2EDuration="21.369302098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:14:54.328943774 +0000 UTC m=+5.556854949" watchObservedRunningTime="2026-07-02 13:15:11.369302098 +0000 UTC m=+22.597213272"
Jul 02 13:15:11.398555 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:11.397435    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6rfcq" podStartSLOduration=3.671669605 podStartE2EDuration="22.397404916s" totalImagesPullingTime="18.725735311s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:11.396183745 +0000 UTC m=+22.624094919" watchObservedRunningTime="2026-07-02 13:15:11.397404916 +0000 UTC m=+22.625316084"
Jul 02 13:15:11.398555 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:11.397645    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hzspc" podStartSLOduration=3.686141353 podStartE2EDuration="22.397618521s" totalImagesPullingTime="18.711477168s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:11.370159874 +0000 UTC m=+22.598071049" watchObservedRunningTime="2026-07-02 13:15:11.397618521 +0000 UTC m=+22.625529697"
Jul 02 13:15:11.466543 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:11.466492    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9pvbh" podStartSLOduration=3.759557545 podStartE2EDuration="22.466470102s" totalImagesPullingTime="18.706912557s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:11.423515246 +0000 UTC m=+22.651426419" watchObservedRunningTime="2026-07-02 13:15:11.466470102 +0000 UTC m=+22.694381277"
Jul 02 13:15:11.489250 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:11.489194    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7jfsq" podStartSLOduration=3.784186309 podStartE2EDuration="22.489173485s" totalImagesPullingTime="18.704987176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:11.488932745 +0000 UTC m=+22.716843920" watchObservedRunningTime="2026-07-02 13:15:11.489173485 +0000 UTC m=+22.717084663"
Jul 02 13:15:11.588629 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:11.588435    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:11.685843 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:11.685735    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:11.799694 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:11.799667    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:11.995192 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:11.995109    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:12.360611 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:12.360516    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9m4tg" event={"ID":"dc9c602d-4c56-4fe4-b1d7-5ef367ca6382","Type":"ContainerStarted","Data":"eb3d9226bfb506612d9f8af79a554bf76aeef9da7e7d245bdd242b51f42e7711"}
Jul 02 13:15:12.363713 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:12.363690    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 13:15:12.364211 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:12.364180    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jlkds" event={"ID":"f0256660-1798-4dba-a311-3705cb73c9ca","Type":"ContainerStarted","Data":"e980dc97900aad66343e637303485b46ab0e1604f4d065da4e05a5fb4b534bac"}
Jul 02 13:15:12.364333 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:12.364254    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jlkds" event={"ID":"f0256660-1798-4dba-a311-3705cb73c9ca","Type":"ContainerStarted","Data":"47339c200170881dbb3b10a821ad324291ce5dd57c530c4832ac2df68cd41055"}
Jul 02 13:15:12.389293 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:12.389248    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2hwvd" podStartSLOduration=4.61982538 podStartE2EDuration="23.389232225s" totalImagesPullingTime="18.769406845s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:11.518480134 +0000 UTC m=+22.746391307" watchObservedRunningTime="2026-07-02 13:15:12.389232225 +0000 UTC m=+23.617143399"
Jul 02 13:15:12.389501 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:12.389427    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9m4tg" podStartSLOduration=4.676562206 podStartE2EDuration="23.389406356s" totalImagesPullingTime="18.71284415s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:12.388868613 +0000 UTC m=+23.616779787" watchObservedRunningTime="2026-07-02 13:15:12.389406356 +0000 UTC m=+23.617317523"
Jul 02 13:15:12.425036 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:12.425000    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7jfsq"
Jul 02 13:15:12.432051 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:12.432020    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7jfsq"
Jul 02 13:15:12.432726 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:12.432707    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7jfsq"
Jul 02 13:15:12.535685 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:12.535653    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:12.577205 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:12.577183    2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 02 13:15:12.701174 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:12.701135    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:12.919940 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:12.919900    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:13.227870 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:13.227618    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:15:13.227870 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:13.227769    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzl6q" podUID="0cae18b4-8f5b-4ff7-a73f-fc668af0a663"
Jul 02 13:15:13.230743 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:13.228829    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:15:13.230743 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:13.228872    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:15:13.230743 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:13.229004    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fpt5" podUID="bfa11a9f-ca92-4214-8e84-0b9c48a8337c"
Jul 02 13:15:13.230743 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:13.229179    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx24z" podUID="159f78cc-1fe0-40ca-a0cd-92254294389c"
Jul 02 13:15:13.234588 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:13.234456    2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-02T13:15:12.577199725Z","UUID":"5bf74fa2-12d0-498e-8236-6e3c76071209","Handler":null,"Name":"","Endpoint":""}
Jul 02 13:15:13.236005 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:13.235982    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 02 13:15:13.236145 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:13.236028    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 02 13:15:13.369038 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:13.368885    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67gtm" event={"ID":"dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2","Type":"ContainerStarted","Data":"8ebf905fe40bb815b1524d96e52721a706100c0f65f289b1714303d1292b0dc8"}
Jul 02 13:15:13.370018 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:13.369997    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7jfsq"
Jul 02 13:15:13.503215 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:13.503098    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:13.819813 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:13.819783    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:13.972500 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:13.972472    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:14.374053 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:14.374026    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 13:15:14.374513 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:14.374427    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jlkds" event={"ID":"f0256660-1798-4dba-a311-3705cb73c9ca","Type":"ContainerStarted","Data":"5aa1d06051195e6c94d85acfc375f8026cb868b85a43d4b6d8ae016ff86c4ade"}
Jul 02 13:15:14.376812 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:14.376772    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67gtm" event={"ID":"dda7049b-02ce-4b45-8ffe-18a5ffe9b6c2","Type":"ContainerStarted","Data":"abb5a049e5de6331c3ecec81b6f23ec5f4f83db03072c59c6c6ada6796137291"}
Jul 02 13:15:14.400225 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:14.400168    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-67gtm" podStartSLOduration=4.155219773 podStartE2EDuration="25.400150323s" totalImagesPullingTime="21.24493055s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:14.398665319 +0000 UTC m=+25.626576495" watchObservedRunningTime="2026-07-02 13:15:14.400150323 +0000 UTC m=+25.628061499"
Jul 02 13:15:14.517675 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:14.517638    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:14.618057 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:14.618021    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:15.228276 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:15.228049    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:15:15.228502 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:15.228089    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:15:15.228502 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:15.228109    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:15:15.228502 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:15.228407    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzl6q" podUID="0cae18b4-8f5b-4ff7-a73f-fc668af0a663"
Jul 02 13:15:15.228674 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:15.228511    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fpt5" podUID="bfa11a9f-ca92-4214-8e84-0b9c48a8337c"
Jul 02 13:15:15.228674 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:15.228602    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx24z" podUID="159f78cc-1fe0-40ca-a0cd-92254294389c"
Jul 02 13:15:15.272038 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:15.272004    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:15.419543 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:15.419508    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:15.982326 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:15.982290    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:16.224517 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:16.224313    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:16.324381 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:16.324255    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:16.384310 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:16.384276    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 13:15:16.384690 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:16.384663    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jlkds" event={"ID":"f0256660-1798-4dba-a311-3705cb73c9ca","Type":"ContainerStarted","Data":"c554a86f32eb0e918a25abe6d12e8cba5c95693b394cf25afaee13eea50fa82b"}
Jul 02 13:15:16.384895 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:16.384869    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:15:16.385120 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:16.385104    2583 scope.go:117] "RemoveContainer" containerID="e2746b9b4e241dc8f1b6788aa49b2dcc5ac29397f36f6da2d22d76d52e12b74a"
Jul 02 13:15:16.386645 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:16.386619    2583 generic.go:356] "Generic (PLEG): container finished" podID="249b4a4b-a5eb-4bc1-8311-1c330010730d" containerID="0d16e6188f2e2342941581b1dffbc4ae2a8b33786f898118d737593e41353e82" exitCode=0
Jul 02 13:15:16.386747 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:16.386651    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjbsm" event={"ID":"249b4a4b-a5eb-4bc1-8311-1c330010730d","Type":"ContainerDied","Data":"0d16e6188f2e2342941581b1dffbc4ae2a8b33786f898118d737593e41353e82"}
Jul 02 13:15:16.401576 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:16.401551    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:15:17.203919 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:17.203866    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:17.231671 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:17.231595    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:15:17.231824 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:17.231746    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzl6q" podUID="0cae18b4-8f5b-4ff7-a73f-fc668af0a663"
Jul 02 13:15:17.232164 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:17.232142    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:15:17.232268 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:17.232247    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fpt5" podUID="bfa11a9f-ca92-4214-8e84-0b9c48a8337c"
Jul 02 13:15:17.232328 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:17.232315    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:15:17.232418 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:17.232380    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx24z" podUID="159f78cc-1fe0-40ca-a0cd-92254294389c"
Jul 02 13:15:17.395220 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:17.395192    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 13:15:17.395567 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:17.395546    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jlkds" event={"ID":"f0256660-1798-4dba-a311-3705cb73c9ca","Type":"ContainerStarted","Data":"e4a303ffe675a8c64c2f7f0e8a69230a90b5725533da9fa9e03df45e60f7f07c"}
Jul 02 13:15:17.395636 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:17.395623    2583 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 02 13:15:17.395774 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:17.395753    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:15:17.410408 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:17.410373    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:15:17.468984 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:17.468926    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jlkds" podStartSLOduration=9.692034433 podStartE2EDuration="28.468908703s" totalImagesPullingTime="18.77687427s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:17.467669406 +0000 UTC m=+28.695580581" watchObservedRunningTime="2026-07-02 13:15:17.468908703 +0000 UTC m=+28.696819877"
Jul 02 13:15:18.167572 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:18.167533    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tx24z"]
Jul 02 13:15:18.167791 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:18.167676    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:15:18.167839 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:18.167780    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx24z" podUID="159f78cc-1fe0-40ca-a0cd-92254294389c"
Jul 02 13:15:18.169058 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:18.169023    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xzl6q"]
Jul 02 13:15:18.169193 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:18.169168    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:15:18.169318 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:18.169273    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzl6q" podUID="0cae18b4-8f5b-4ff7-a73f-fc668af0a663"
Jul 02 13:15:18.169688 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:18.169668    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9fpt5"]
Jul 02 13:15:18.169796 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:18.169781    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:15:18.169914 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:18.169878    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fpt5" podUID="bfa11a9f-ca92-4214-8e84-0b9c48a8337c"
Jul 02 13:15:18.400260 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:18.400228    2583 generic.go:356] "Generic (PLEG): container finished" podID="249b4a4b-a5eb-4bc1-8311-1c330010730d" containerID="82d0b2330a5cc6d51f50cac26c3faaa7802ab77c6508b7427d9709f6b23c6847" exitCode=0
Jul 02 13:15:18.400992 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:18.400308    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjbsm" event={"ID":"249b4a4b-a5eb-4bc1-8311-1c330010730d","Type":"ContainerDied","Data":"82d0b2330a5cc6d51f50cac26c3faaa7802ab77c6508b7427d9709f6b23c6847"}
Jul 02 13:15:18.400992 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:18.400430    2583 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 02 13:15:18.609922 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:18.609830    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:15:19.231830 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:19.231801    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:15:19.232018 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:19.231920    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzl6q" podUID="0cae18b4-8f5b-4ff7-a73f-fc668af0a663"
Jul 02 13:15:19.232018 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:19.231925    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:15:19.232109 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:19.232015    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx24z" podUID="159f78cc-1fe0-40ca-a0cd-92254294389c"
Jul 02 13:15:20.228716 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:20.228522    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:15:20.229134 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:20.228793    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fpt5" podUID="bfa11a9f-ca92-4214-8e84-0b9c48a8337c"
Jul 02 13:15:20.407720 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:20.407633    2583 generic.go:356] "Generic (PLEG): container finished" podID="249b4a4b-a5eb-4bc1-8311-1c330010730d" containerID="37f2fc671773a2075c50a7fd76648d8e87a7af35c3c382f1a2a62b4b8e41c915" exitCode=0
Jul 02 13:15:20.407720 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:20.407709    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjbsm" event={"ID":"249b4a4b-a5eb-4bc1-8311-1c330010730d","Type":"ContainerDied","Data":"37f2fc671773a2075c50a7fd76648d8e87a7af35c3c382f1a2a62b4b8e41c915"}
Jul 02 13:15:20.424280 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:20.424193    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-jlkds" podUID="f0256660-1798-4dba-a311-3705cb73c9ca" containerName="ovnkube-controller" probeResult="failure" output=""
Jul 02 13:15:20.532141 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:20.532108    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:20.925108 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:20.925061    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:21.115639 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:21.115605    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:21.212562 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:21.212479    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:21.228269 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:21.228241    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:15:21.228269 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:21.228255    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:15:21.228526 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:21.228373    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzl6q" podUID="0cae18b4-8f5b-4ff7-a73f-fc668af0a663"
Jul 02 13:15:21.228526 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:21.228481    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx24z" podUID="159f78cc-1fe0-40ca-a0cd-92254294389c"
Jul 02 13:15:21.341447 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:21.341364    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:22.017314 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:22.017278    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:22.228423 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:22.228381    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:15:22.228592 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:22.228517    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fpt5" podUID="bfa11a9f-ca92-4214-8e84-0b9c48a8337c"
Jul 02 13:15:22.274095 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:22.274005    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:22.801625 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:22.801590    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:22.874424 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:22.874377    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:22.921692 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:22.921649    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cae18b4-8f5b-4ff7-a73f-fc668af0a663-metrics-certs\") pod \"network-metrics-daemon-xzl6q\" (UID: \"0cae18b4-8f5b-4ff7-a73f-fc668af0a663\") " pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:15:22.921860 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:22.921729    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bfa11a9f-ca92-4214-8e84-0b9c48a8337c-original-pull-secret\") pod \"global-pull-secret-syncer-9fpt5\" (UID: \"bfa11a9f-ca92-4214-8e84-0b9c48a8337c\") " pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:15:22.921860 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:22.921798    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:15:22.921860 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:22.921850    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:15:22.921973 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:22.921880    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cae18b4-8f5b-4ff7-a73f-fc668af0a663-metrics-certs podName:0cae18b4-8f5b-4ff7-a73f-fc668af0a663 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:54.921859066 +0000 UTC m=+66.149770223 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cae18b4-8f5b-4ff7-a73f-fc668af0a663-metrics-certs") pod "network-metrics-daemon-xzl6q" (UID: "0cae18b4-8f5b-4ff7-a73f-fc668af0a663") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:15:22.921973 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:22.921901    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfa11a9f-ca92-4214-8e84-0b9c48a8337c-original-pull-secret podName:bfa11a9f-ca92-4214-8e84-0b9c48a8337c nodeName:}" failed. No retries permitted until 2026-07-02 13:15:54.921890237 +0000 UTC m=+66.149801394 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bfa11a9f-ca92-4214-8e84-0b9c48a8337c-original-pull-secret") pod "global-pull-secret-syncer-9fpt5" (UID: "bfa11a9f-ca92-4214-8e84-0b9c48a8337c") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:15:23.087029 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:23.086949    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:23.123225 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:23.123185    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2r82\" (UniqueName: \"kubernetes.io/projected/159f78cc-1fe0-40ca-a0cd-92254294389c-kube-api-access-s2r82\") pod \"network-check-target-tx24z\" (UID: \"159f78cc-1fe0-40ca-a0cd-92254294389c\") " pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:15:23.123406 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:23.123312    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:15:23.123406 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:23.123326    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:15:23.123406 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:23.123335    2583 projected.go:196] Error preparing data for projected volume kube-api-access-s2r82 for pod openshift-network-diagnostics/network-check-target-tx24z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:15:23.123517 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:23.123412    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/159f78cc-1fe0-40ca-a0cd-92254294389c-kube-api-access-s2r82 podName:159f78cc-1fe0-40ca-a0cd-92254294389c nodeName:}" failed. No retries permitted until 2026-07-02 13:15:55.123382029 +0000 UTC m=+66.351293180 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-s2r82" (UniqueName: "kubernetes.io/projected/159f78cc-1fe0-40ca-a0cd-92254294389c-kube-api-access-s2r82") pod "network-check-target-tx24z" (UID: "159f78cc-1fe0-40ca-a0cd-92254294389c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:15:23.228306 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:23.228267    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:15:23.228483 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:23.228412    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx24z" podUID="159f78cc-1fe0-40ca-a0cd-92254294389c"
Jul 02 13:15:23.228483 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:23.228462    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:15:23.228611 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:23.228594    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzl6q" podUID="0cae18b4-8f5b-4ff7-a73f-fc668af0a663"
Jul 02 13:15:23.617007 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:23.616971    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:24.121760 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:24.121728    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:24.227656 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:24.227623    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:15:24.227836 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:24.227753    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fpt5" podUID="bfa11a9f-ca92-4214-8e84-0b9c48a8337c"
Jul 02 13:15:24.417522 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:24.417360    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:24.601492 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:24.601451    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:24.702591 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:24.702479    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:25.001915 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:25.001828    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:25.207938 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:25.207906    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:25.227979 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:25.227945    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:15:25.228153 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:25.227952    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:15:25.228153 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:25.228072    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx24z" podUID="159f78cc-1fe0-40ca-a0cd-92254294389c"
Jul 02 13:15:25.228270 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:25.228180    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzl6q" podUID="0cae18b4-8f5b-4ff7-a73f-fc668af0a663"
Jul 02 13:15:25.821349 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:25.821315    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:25.922219 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:25.922186    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:26.022031 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:26.021945    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:26.227837 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:26.227810    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:15:26.228203 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:26.227911    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fpt5" podUID="bfa11a9f-ca92-4214-8e84-0b9c48a8337c"
Jul 02 13:15:26.424784 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:26.424744    2583 generic.go:356] "Generic (PLEG): container finished" podID="249b4a4b-a5eb-4bc1-8311-1c330010730d" containerID="bbb4acbb0b471a817b45398cd80fa5c54942bd5762dd750e85b17de8e918f942" exitCode=0
Jul 02 13:15:26.424784 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:26.424784    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjbsm" event={"ID":"249b4a4b-a5eb-4bc1-8311-1c330010730d","Type":"ContainerDied","Data":"bbb4acbb0b471a817b45398cd80fa5c54942bd5762dd750e85b17de8e918f942"}
Jul 02 13:15:26.513209 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:26.513179    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:26.612973 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:26.612938    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:26.996892 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:26.996852    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:27.228542 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:27.228509    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:15:27.229056 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:27.228516    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:15:27.229056 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:27.228611    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx24z" podUID="159f78cc-1fe0-40ca-a0cd-92254294389c"
Jul 02 13:15:27.229056 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:27.228726    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzl6q" podUID="0cae18b4-8f5b-4ff7-a73f-fc668af0a663"
Jul 02 13:15:27.396796 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:27.396761    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:27.429709 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:27.429677    2583 generic.go:356] "Generic (PLEG): container finished" podID="249b4a4b-a5eb-4bc1-8311-1c330010730d" containerID="4ac60c77399eaa4cca12ddaabe9a558d13333ae201e0a221d6fe08b5bf24e838" exitCode=0
Jul 02 13:15:27.429890 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:27.429749    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjbsm" event={"ID":"249b4a4b-a5eb-4bc1-8311-1c330010730d","Type":"ContainerDied","Data":"4ac60c77399eaa4cca12ddaabe9a558d13333ae201e0a221d6fe08b5bf24e838"}
Jul 02 13:15:27.791116 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:27.790905    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:28.008048 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:28.007953    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:28.159961 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:28.159933    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:28.228312 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:28.228284    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:15:28.228456 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:28.228374    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fpt5" podUID="bfa11a9f-ca92-4214-8e84-0b9c48a8337c"
Jul 02 13:15:28.406170 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:28.406141    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:28.434897 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:28.434868    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjbsm" event={"ID":"249b4a4b-a5eb-4bc1-8311-1c330010730d","Type":"ContainerStarted","Data":"56ec8556649c187dfdda7c8405f1344381ff54550173a65f7ef46eda33cd56ee"}
Jul 02 13:15:28.461469 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:28.461423    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fjbsm" podStartSLOduration=7.780610824 podStartE2EDuration="39.461401622s" totalImagesPullingTime="31.680790798s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:28.461123737 +0000 UTC m=+39.689034924" watchObservedRunningTime="2026-07-02 13:15:28.461401622 +0000 UTC m=+39.689312788"
Jul 02 13:15:29.014646 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:29.014610    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:29.165837 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:29.165785    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:29.228850 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:29.228820    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:15:29.229020 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:29.228934    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:15:29.229077 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:29.228933    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx24z" podUID="159f78cc-1fe0-40ca-a0cd-92254294389c"
Jul 02 13:15:29.229125 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:29.229070    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzl6q" podUID="0cae18b4-8f5b-4ff7-a73f-fc668af0a663"
Jul 02 13:15:29.242258 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:29.242235    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:29.626892 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:29.626863    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:29.825021 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:29.824993    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:30.204066 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:30.204031    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:30.228273 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:30.228241    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:15:30.228380 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:30.228356    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fpt5" podUID="bfa11a9f-ca92-4214-8e84-0b9c48a8337c"
Jul 02 13:15:30.805570 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:30.805535    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:31.206880 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:31.206852    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:31.228512 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:31.228483    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:15:31.228639 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:31.228482    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:15:31.228639 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:31.228591    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx24z" podUID="159f78cc-1fe0-40ca-a0cd-92254294389c"
Jul 02 13:15:31.228744 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:31.228687    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzl6q" podUID="0cae18b4-8f5b-4ff7-a73f-fc668af0a663"
Jul 02 13:15:31.410191 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:31.410163    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:31.609092 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:31.609005    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:32.228458 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:32.228427    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:15:32.228886 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:32.228533    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fpt5" podUID="bfa11a9f-ca92-4214-8e84-0b9c48a8337c"
Jul 02 13:15:33.228481 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:33.228451    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:15:33.228848 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:33.228556    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx24z" podUID="159f78cc-1fe0-40ca-a0cd-92254294389c"
Jul 02 13:15:33.228848 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:33.228609    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:15:33.228848 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:33.228726    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzl6q" podUID="0cae18b4-8f5b-4ff7-a73f-fc668af0a663"
Jul 02 13:15:34.227802 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:34.227771    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:15:34.227978 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:34.227869    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fpt5" podUID="bfa11a9f-ca92-4214-8e84-0b9c48a8337c"
Jul 02 13:15:34.272278 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:34.272242    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:34.432572 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:34.432541    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:35.055422 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:35.055375    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:35.228344 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:35.228303    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:15:35.228524 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:35.228474    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xzl6q" podUID="0cae18b4-8f5b-4ff7-a73f-fc668af0a663"
Jul 02 13:15:35.228572 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:35.228523    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:15:35.231521 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:35.228870    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tx24z" podUID="159f78cc-1fe0-40ca-a0cd-92254294389c"
Jul 02 13:15:36.029986 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:36.029950    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:36.227510 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:36.227478    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:15:36.227657 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:36.227588    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9fpt5" podUID="bfa11a9f-ca92-4214-8e84-0b9c48a8337c"
Jul 02 13:15:36.253247 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:36.253216    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:36.412347 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:36.412318    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:37.061815 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.061790    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-143.ec2.internal" event="NodeReady"
Jul 02 13:15:37.062174 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.061937    2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 02 13:15:37.135360 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.135328    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tkws4"]
Jul 02 13:15:37.169808 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.169767    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-n5cmj"]
Jul 02 13:15:37.169984 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.169896    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tkws4"
Jul 02 13:15:37.172713 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.172690    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h5pfr\""
Jul 02 13:15:37.172864 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.172815    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 02 13:15:37.173141 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.173127    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 02 13:15:37.189401 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.189350    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tkws4"]
Jul 02 13:15:37.189549 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.189423    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n5cmj"]
Jul 02 13:15:37.189549 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.189492    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n5cmj"
Jul 02 13:15:37.192136 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.192103    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 02 13:15:37.192260 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.192230    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 02 13:15:37.192260 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.192254    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 02 13:15:37.192374 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.192304    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 02 13:15:37.192574 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.192558    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8kq9n\""
Jul 02 13:15:37.227625 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.227591    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:15:37.227813 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.227793    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:15:37.234228 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.231824    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 13:15:37.234228 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.232005    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zjxr6\""
Jul 02 13:15:37.234228 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.232230    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-drw6p\""
Jul 02 13:15:37.234228 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.233020    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dd8n6"]
Jul 02 13:15:37.234834 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.234818    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 13:15:37.234878 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.234855    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 13:15:37.261474 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.261435    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dd8n6"]
Jul 02 13:15:37.261636 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.261566    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dd8n6"
Jul 02 13:15:37.265007 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.264988    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fkpd6\""
Jul 02 13:15:37.265410 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.265379    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 02 13:15:37.265624 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.265596    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 02 13:15:37.265828 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.265814    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 02 13:15:37.326749 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.326665    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3810cfce-45d5-47b2-950c-a17f109aa7ac-crio-socket\") pod \"insights-runtime-extractor-n5cmj\" (UID: \"3810cfce-45d5-47b2-950c-a17f109aa7ac\") " pod="openshift-insights/insights-runtime-extractor-n5cmj"
Jul 02 13:15:37.326749 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.326697    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3810cfce-45d5-47b2-950c-a17f109aa7ac-data-volume\") pod \"insights-runtime-extractor-n5cmj\" (UID: \"3810cfce-45d5-47b2-950c-a17f109aa7ac\") " pod="openshift-insights/insights-runtime-extractor-n5cmj"
Jul 02 13:15:37.326749 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.326723    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/74475ceb-1fd0-4b3d-a5f4-60c91249a748-config-volume\") pod \"dns-default-tkws4\" (UID: \"74475ceb-1fd0-4b3d-a5f4-60c91249a748\") " pod="openshift-dns/dns-default-tkws4"
Jul 02 13:15:37.326749 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.326742    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3810cfce-45d5-47b2-950c-a17f109aa7ac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n5cmj\" (UID: \"3810cfce-45d5-47b2-950c-a17f109aa7ac\") " pod="openshift-insights/insights-runtime-extractor-n5cmj"
Jul 02 13:15:37.326991 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.326821    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74475ceb-1fd0-4b3d-a5f4-60c91249a748-tmp-dir\") pod \"dns-default-tkws4\" (UID: \"74475ceb-1fd0-4b3d-a5f4-60c91249a748\") " pod="openshift-dns/dns-default-tkws4"
Jul 02 13:15:37.326991 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.326838    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3810cfce-45d5-47b2-950c-a17f109aa7ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n5cmj\" (UID: \"3810cfce-45d5-47b2-950c-a17f109aa7ac\") " pod="openshift-insights/insights-runtime-extractor-n5cmj"
Jul 02 13:15:37.326991 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.326857    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f877n\" (UniqueName: \"kubernetes.io/projected/3810cfce-45d5-47b2-950c-a17f109aa7ac-kube-api-access-f877n\") pod \"insights-runtime-extractor-n5cmj\" (UID: \"3810cfce-45d5-47b2-950c-a17f109aa7ac\") " pod="openshift-insights/insights-runtime-extractor-n5cmj"
Jul 02 13:15:37.326991 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.326875    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssghk\" (UniqueName: \"kubernetes.io/projected/74475ceb-1fd0-4b3d-a5f4-60c91249a748-kube-api-access-ssghk\") pod \"dns-default-tkws4\" (UID: \"74475ceb-1fd0-4b3d-a5f4-60c91249a748\") " pod="openshift-dns/dns-default-tkws4"
Jul 02 13:15:37.326991 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.326894    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74475ceb-1fd0-4b3d-a5f4-60c91249a748-metrics-tls\") pod \"dns-default-tkws4\" (UID: \"74475ceb-1fd0-4b3d-a5f4-60c91249a748\") " pod="openshift-dns/dns-default-tkws4"
Jul 02 13:15:37.355582 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.355550    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:37.427876 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.427840    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74475ceb-1fd0-4b3d-a5f4-60c91249a748-tmp-dir\") pod \"dns-default-tkws4\" (UID: \"74475ceb-1fd0-4b3d-a5f4-60c91249a748\") " pod="openshift-dns/dns-default-tkws4"
Jul 02 13:15:37.427876 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.427877    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3810cfce-45d5-47b2-950c-a17f109aa7ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n5cmj\" (UID: \"3810cfce-45d5-47b2-950c-a17f109aa7ac\") " pod="openshift-insights/insights-runtime-extractor-n5cmj"
Jul 02 13:15:37.428105 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.427904    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f877n\" (UniqueName: \"kubernetes.io/projected/3810cfce-45d5-47b2-950c-a17f109aa7ac-kube-api-access-f877n\") pod \"insights-runtime-extractor-n5cmj\" (UID: \"3810cfce-45d5-47b2-950c-a17f109aa7ac\") " pod="openshift-insights/insights-runtime-extractor-n5cmj"
Jul 02 13:15:37.428105 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.427930    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc66f32c-20f9-44d2-b58e-5e3b8ede1958-cert\") pod \"ingress-canary-dd8n6\" (UID: \"bc66f32c-20f9-44d2-b58e-5e3b8ede1958\") " pod="openshift-ingress-canary/ingress-canary-dd8n6"
Jul 02 13:15:37.428105 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.427959    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssghk\" (UniqueName: \"kubernetes.io/projected/74475ceb-1fd0-4b3d-a5f4-60c91249a748-kube-api-access-ssghk\") pod \"dns-default-tkws4\" (UID: \"74475ceb-1fd0-4b3d-a5f4-60c91249a748\") " pod="openshift-dns/dns-default-tkws4"
Jul 02 13:15:37.428105 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.427994    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74475ceb-1fd0-4b3d-a5f4-60c91249a748-metrics-tls\") pod \"dns-default-tkws4\" (UID: \"74475ceb-1fd0-4b3d-a5f4-60c91249a748\") " pod="openshift-dns/dns-default-tkws4"
Jul 02 13:15:37.428105 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.428017    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqwgw\" (UniqueName: \"kubernetes.io/projected/bc66f32c-20f9-44d2-b58e-5e3b8ede1958-kube-api-access-bqwgw\") pod \"ingress-canary-dd8n6\" (UID: \"bc66f32c-20f9-44d2-b58e-5e3b8ede1958\") " pod="openshift-ingress-canary/ingress-canary-dd8n6"
Jul 02 13:15:37.428105 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.428092    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3810cfce-45d5-47b2-950c-a17f109aa7ac-crio-socket\") pod \"insights-runtime-extractor-n5cmj\" (UID: \"3810cfce-45d5-47b2-950c-a17f109aa7ac\") " pod="openshift-insights/insights-runtime-extractor-n5cmj"
Jul 02 13:15:37.428363 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.428134    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3810cfce-45d5-47b2-950c-a17f109aa7ac-data-volume\") pod \"insights-runtime-extractor-n5cmj\" (UID: \"3810cfce-45d5-47b2-950c-a17f109aa7ac\") " pod="openshift-insights/insights-runtime-extractor-n5cmj"
Jul 02 13:15:37.428363 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.428203    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/74475ceb-1fd0-4b3d-a5f4-60c91249a748-config-volume\") pod \"dns-default-tkws4\" (UID: \"74475ceb-1fd0-4b3d-a5f4-60c91249a748\") " pod="openshift-dns/dns-default-tkws4"
Jul 02 13:15:37.428363 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.428224    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74475ceb-1fd0-4b3d-a5f4-60c91249a748-tmp-dir\") pod \"dns-default-tkws4\" (UID: \"74475ceb-1fd0-4b3d-a5f4-60c91249a748\") " pod="openshift-dns/dns-default-tkws4"
Jul 02 13:15:37.428363 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.428234    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3810cfce-45d5-47b2-950c-a17f109aa7ac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n5cmj\" (UID: \"3810cfce-45d5-47b2-950c-a17f109aa7ac\") " pod="openshift-insights/insights-runtime-extractor-n5cmj"
Jul 02 13:15:37.428363 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.428348    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3810cfce-45d5-47b2-950c-a17f109aa7ac-crio-socket\") pod \"insights-runtime-extractor-n5cmj\" (UID: \"3810cfce-45d5-47b2-950c-a17f109aa7ac\") " pod="openshift-insights/insights-runtime-extractor-n5cmj"
Jul 02 13:15:37.428709 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.428682    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3810cfce-45d5-47b2-950c-a17f109aa7ac-data-volume\") pod \"insights-runtime-extractor-n5cmj\" (UID: \"3810cfce-45d5-47b2-950c-a17f109aa7ac\") " pod="openshift-insights/insights-runtime-extractor-n5cmj"
Jul 02 13:15:37.428834 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.428809    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/74475ceb-1fd0-4b3d-a5f4-60c91249a748-config-volume\") pod \"dns-default-tkws4\" (UID: \"74475ceb-1fd0-4b3d-a5f4-60c91249a748\") " pod="openshift-dns/dns-default-tkws4"
Jul 02 13:15:37.428834 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.428817    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3810cfce-45d5-47b2-950c-a17f109aa7ac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n5cmj\" (UID: \"3810cfce-45d5-47b2-950c-a17f109aa7ac\") " pod="openshift-insights/insights-runtime-extractor-n5cmj"
Jul 02 13:15:37.431890 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.431866    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74475ceb-1fd0-4b3d-a5f4-60c91249a748-metrics-tls\") pod \"dns-default-tkws4\" (UID: \"74475ceb-1fd0-4b3d-a5f4-60c91249a748\") " pod="openshift-dns/dns-default-tkws4"
Jul 02 13:15:37.431992 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.431932    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3810cfce-45d5-47b2-950c-a17f109aa7ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n5cmj\" (UID: \"3810cfce-45d5-47b2-950c-a17f109aa7ac\") " pod="openshift-insights/insights-runtime-extractor-n5cmj"
Jul 02 13:15:37.442639 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.442615    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f877n\" (UniqueName: \"kubernetes.io/projected/3810cfce-45d5-47b2-950c-a17f109aa7ac-kube-api-access-f877n\") pod \"insights-runtime-extractor-n5cmj\" (UID: \"3810cfce-45d5-47b2-950c-a17f109aa7ac\") " pod="openshift-insights/insights-runtime-extractor-n5cmj"
Jul 02 13:15:37.455091 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.455062    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssghk\" (UniqueName: \"kubernetes.io/projected/74475ceb-1fd0-4b3d-a5f4-60c91249a748-kube-api-access-ssghk\") pod \"dns-default-tkws4\" (UID: \"74475ceb-1fd0-4b3d-a5f4-60c91249a748\") " pod="openshift-dns/dns-default-tkws4"
Jul 02 13:15:37.462073 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.462048    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:37.478708 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.478686    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tkws4"
Jul 02 13:15:37.497873 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.497844    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n5cmj"
Jul 02 13:15:37.529551 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.529525    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqwgw\" (UniqueName: \"kubernetes.io/projected/bc66f32c-20f9-44d2-b58e-5e3b8ede1958-kube-api-access-bqwgw\") pod \"ingress-canary-dd8n6\" (UID: \"bc66f32c-20f9-44d2-b58e-5e3b8ede1958\") " pod="openshift-ingress-canary/ingress-canary-dd8n6"
Jul 02 13:15:37.529680 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.529604    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc66f32c-20f9-44d2-b58e-5e3b8ede1958-cert\") pod \"ingress-canary-dd8n6\" (UID: \"bc66f32c-20f9-44d2-b58e-5e3b8ede1958\") " pod="openshift-ingress-canary/ingress-canary-dd8n6"
Jul 02 13:15:37.532086 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.532058    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc66f32c-20f9-44d2-b58e-5e3b8ede1958-cert\") pod \"ingress-canary-dd8n6\" (UID: \"bc66f32c-20f9-44d2-b58e-5e3b8ede1958\") " pod="openshift-ingress-canary/ingress-canary-dd8n6"
Jul 02 13:15:37.540424 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.540377    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqwgw\" (UniqueName: \"kubernetes.io/projected/bc66f32c-20f9-44d2-b58e-5e3b8ede1958-kube-api-access-bqwgw\") pod \"ingress-canary-dd8n6\" (UID: \"bc66f32c-20f9-44d2-b58e-5e3b8ede1958\") " pod="openshift-ingress-canary/ingress-canary-dd8n6"
Jul 02 13:15:37.570205 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.570173    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dd8n6"
Jul 02 13:15:37.728506 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.728419    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tkws4"]
Jul 02 13:15:37.729179 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.729102    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n5cmj"]
Jul 02 13:15:37.730812 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:15:37.730786    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3810cfce_45d5_47b2_950c_a17f109aa7ac.slice/crio-8ea11e3e525c24ba61a31a06411ecc14460dc4a5ae65339e16aecd342a3d40f8 WatchSource:0}: Error finding container 8ea11e3e525c24ba61a31a06411ecc14460dc4a5ae65339e16aecd342a3d40f8: Status 404 returned error can't find the container with id 8ea11e3e525c24ba61a31a06411ecc14460dc4a5ae65339e16aecd342a3d40f8
Jul 02 13:15:37.731613 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:15:37.731591    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74475ceb_1fd0_4b3d_a5f4_60c91249a748.slice/crio-99362574be9d10b7dda85c53df26c90df4313039ad41a8e7d7309618b20fede7 WatchSource:0}: Error finding container 99362574be9d10b7dda85c53df26c90df4313039ad41a8e7d7309618b20fede7: Status 404 returned error can't find the container with id 99362574be9d10b7dda85c53df26c90df4313039ad41a8e7d7309618b20fede7
Jul 02 13:15:37.731730 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.731708    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:37.759302 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.759268    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dd8n6"]
Jul 02 13:15:37.762331 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:15:37.762308    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc66f32c_20f9_44d2_b58e_5e3b8ede1958.slice/crio-be488df72f34c366b1a1708f4ff9d4a60cda591735e211128c89d5673e5995a4 WatchSource:0}: Error finding container be488df72f34c366b1a1708f4ff9d4a60cda591735e211128c89d5673e5995a4: Status 404 returned error can't find the container with id be488df72f34c366b1a1708f4ff9d4a60cda591735e211128c89d5673e5995a4
Jul 02 13:15:37.857295 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:37.857208    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:38.227921 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:38.227894    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:15:38.231046 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:38.231024    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 13:15:38.458179 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:38.458140    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dd8n6" event={"ID":"bc66f32c-20f9-44d2-b58e-5e3b8ede1958","Type":"ContainerStarted","Data":"be488df72f34c366b1a1708f4ff9d4a60cda591735e211128c89d5673e5995a4"}
Jul 02 13:15:38.459233 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:38.459213    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n5cmj" event={"ID":"3810cfce-45d5-47b2-950c-a17f109aa7ac","Type":"ContainerStarted","Data":"8ea11e3e525c24ba61a31a06411ecc14460dc4a5ae65339e16aecd342a3d40f8"}
Jul 02 13:15:38.460235 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:38.460213    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tkws4" event={"ID":"74475ceb-1fd0-4b3d-a5f4-60c91249a748","Type":"ContainerStarted","Data":"99362574be9d10b7dda85c53df26c90df4313039ad41a8e7d7309618b20fede7"}
Jul 02 13:15:42.385159 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:42.385120    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:42.464009 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:42.463971    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:43.546559 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:43.546526    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-78l79"]
Jul 02 13:15:43.632994 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:43.632953    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-78l79"]
Jul 02 13:15:43.633160 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:43.633079    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7b7d896497-78l79"
Jul 02 13:15:43.636368 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:43.636342    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 02 13:15:43.636540 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:43.636449    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 02 13:15:43.637407 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:43.637368    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 02 13:15:43.637543 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:43.637435    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8xbqc\""
Jul 02 13:15:43.637677 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:43.637662    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 02 13:15:43.637770 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:43.637715    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 02 13:15:43.674629 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:43.674592    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/330f7db3-f601-4139-9272-87bc867c9ad1-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-78l79\" (UID: \"330f7db3-f601-4139-9272-87bc867c9ad1\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-78l79"
Jul 02 13:15:43.674807 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:43.674655    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxnng\" (UniqueName: \"kubernetes.io/projected/330f7db3-f601-4139-9272-87bc867c9ad1-kube-api-access-kxnng\") pod \"prometheus-operator-7b7d896497-78l79\" (UID: \"330f7db3-f601-4139-9272-87bc867c9ad1\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-78l79"
Jul 02 13:15:43.674807 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:43.674686    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/330f7db3-f601-4139-9272-87bc867c9ad1-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-78l79\" (UID: \"330f7db3-f601-4139-9272-87bc867c9ad1\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-78l79"
Jul 02 13:15:43.674807 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:43.674709    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/330f7db3-f601-4139-9272-87bc867c9ad1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-78l79\" (UID: \"330f7db3-f601-4139-9272-87bc867c9ad1\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-78l79"
Jul 02 13:15:43.775289 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:43.775248    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxnng\" (UniqueName: \"kubernetes.io/projected/330f7db3-f601-4139-9272-87bc867c9ad1-kube-api-access-kxnng\") pod \"prometheus-operator-7b7d896497-78l79\" (UID: \"330f7db3-f601-4139-9272-87bc867c9ad1\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-78l79"
Jul 02 13:15:43.775289 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:43.775292    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/330f7db3-f601-4139-9272-87bc867c9ad1-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-78l79\" (UID: \"330f7db3-f601-4139-9272-87bc867c9ad1\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-78l79"
Jul 02 13:15:43.775573 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:43.775314    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/330f7db3-f601-4139-9272-87bc867c9ad1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-78l79\" (UID: \"330f7db3-f601-4139-9272-87bc867c9ad1\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-78l79"
Jul 02 13:15:43.775573 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:43.775401    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/330f7db3-f601-4139-9272-87bc867c9ad1-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-78l79\" (UID: \"330f7db3-f601-4139-9272-87bc867c9ad1\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-78l79"
Jul 02 13:15:43.775573 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:43.775490    2583 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 02 13:15:43.775573 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:43.775545    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/330f7db3-f601-4139-9272-87bc867c9ad1-prometheus-operator-tls podName:330f7db3-f601-4139-9272-87bc867c9ad1 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:44.275531056 +0000 UTC m=+55.503442208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/330f7db3-f601-4139-9272-87bc867c9ad1-prometheus-operator-tls") pod "prometheus-operator-7b7d896497-78l79" (UID: "330f7db3-f601-4139-9272-87bc867c9ad1") : secret "prometheus-operator-tls" not found
Jul 02 13:15:43.776026 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:43.775999    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/330f7db3-f601-4139-9272-87bc867c9ad1-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-78l79\" (UID: \"330f7db3-f601-4139-9272-87bc867c9ad1\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-78l79"
Jul 02 13:15:43.777734 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:43.777706    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/330f7db3-f601-4139-9272-87bc867c9ad1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-78l79\" (UID: \"330f7db3-f601-4139-9272-87bc867c9ad1\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-78l79"
Jul 02 13:15:43.789598 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:43.789571    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxnng\" (UniqueName: \"kubernetes.io/projected/330f7db3-f601-4139-9272-87bc867c9ad1-kube-api-access-kxnng\") pod \"prometheus-operator-7b7d896497-78l79\" (UID: \"330f7db3-f601-4139-9272-87bc867c9ad1\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-78l79"
Jul 02 13:15:43.951438 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:43.951399    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:44.279005 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:44.278921    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/330f7db3-f601-4139-9272-87bc867c9ad1-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-78l79\" (UID: \"330f7db3-f601-4139-9272-87bc867c9ad1\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-78l79"
Jul 02 13:15:44.281576 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:44.281548    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/330f7db3-f601-4139-9272-87bc867c9ad1-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-78l79\" (UID: \"330f7db3-f601-4139-9272-87bc867c9ad1\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-78l79"
Jul 02 13:15:44.503225 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:44.503200    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:44.543625 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:44.543594    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7b7d896497-78l79"
Jul 02 13:15:44.709076 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:44.708969    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-78l79"]
Jul 02 13:15:44.714505 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:15:44.714477    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330f7db3_f601_4139_9272_87bc867c9ad1.slice/crio-da222231d18c5185e35e4e0e330e858adc5195e118a7e7aa0f00e742193cf7e3 WatchSource:0}: Error finding container da222231d18c5185e35e4e0e330e858adc5195e118a7e7aa0f00e742193cf7e3: Status 404 returned error can't find the container with id da222231d18c5185e35e4e0e330e858adc5195e118a7e7aa0f00e742193cf7e3
Jul 02 13:15:45.481929 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:45.481890    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n5cmj" event={"ID":"3810cfce-45d5-47b2-950c-a17f109aa7ac","Type":"ContainerStarted","Data":"3b75246063bb29a1cde22a504fae8e92c068064ca269d24b56ef6ebc27889fa8"}
Jul 02 13:15:45.483051 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:45.483029    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-78l79" event={"ID":"330f7db3-f601-4139-9272-87bc867c9ad1","Type":"ContainerStarted","Data":"da222231d18c5185e35e4e0e330e858adc5195e118a7e7aa0f00e742193cf7e3"}
Jul 02 13:15:45.484605 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:45.484588    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tkws4" event={"ID":"74475ceb-1fd0-4b3d-a5f4-60c91249a748","Type":"ContainerStarted","Data":"d366ffe3bf45bf58c75e5c27988f587ba7c0450ac79edc7060ca08b169165ff2"}
Jul 02 13:15:45.484673 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:45.484624    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tkws4" event={"ID":"74475ceb-1fd0-4b3d-a5f4-60c91249a748","Type":"ContainerStarted","Data":"3f200820aa5eadb4b07d934ef233c5a7f132aaa70f95545002c3c41d02f10d43"}
Jul 02 13:15:45.484721 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:45.484704    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tkws4"
Jul 02 13:15:45.485909 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:45.485891    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dd8n6" event={"ID":"bc66f32c-20f9-44d2-b58e-5e3b8ede1958","Type":"ContainerStarted","Data":"7fe26628dcaf6e8e8b6166ef25cae487515214b8577158c4d10434ec08f64ead"}
Jul 02 13:15:45.506834 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:45.506777    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tkws4" podStartSLOduration=1.820980824 podStartE2EDuration="8.506760889s" totalImagesPullingTime="6.685780065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:45.505183348 +0000 UTC m=+56.733094522" watchObservedRunningTime="2026-07-02 13:15:45.506760889 +0000 UTC m=+56.734672064"
Jul 02 13:15:47.326524 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:47.326489    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:47.498330 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:47.498296    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n5cmj" event={"ID":"3810cfce-45d5-47b2-950c-a17f109aa7ac","Type":"ContainerStarted","Data":"9d0ab895d76714c9705d81c67bd79564e97914d459586f67b7e792475cb375d4"}
Jul 02 13:15:47.499938 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:47.499917    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-78l79" event={"ID":"330f7db3-f601-4139-9272-87bc867c9ad1","Type":"ContainerStarted","Data":"a54496d68c13eafa3418c4cd19b1117a2c5cbf10ac509be57f8b54b725071594"}
Jul 02 13:15:47.500034 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:47.499957    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-78l79" event={"ID":"330f7db3-f601-4139-9272-87bc867c9ad1","Type":"ContainerStarted","Data":"e2851b46ca082f487e7f3f9a7fc53036d08fd0a6d4c26a10c05e037e2bab29f6"}
Jul 02 13:15:47.537095 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:47.536971    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dd8n6" podStartSLOduration=3.881615277 podStartE2EDuration="10.536951677s" totalImagesPullingTime="6.6553364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:45.527835484 +0000 UTC m=+56.755746658" watchObservedRunningTime="2026-07-02 13:15:47.536951677 +0000 UTC m=+58.764862853"
Jul 02 13:15:47.537636 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:47.537602    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7b7d896497-78l79" podStartSLOduration=2.288094544 podStartE2EDuration="4.537590701s" totalImagesPullingTime="2.249496157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:47.536888528 +0000 UTC m=+58.764799792" watchObservedRunningTime="2026-07-02 13:15:47.537590701 +0000 UTC m=+58.765501874"
Jul 02 13:15:49.836982 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:49.836932    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:50.074272 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.074236    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:50.232002 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.231968    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:50.280309 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.280278    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-drjzm"]
Jul 02 13:15:50.282342 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.282320    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-845b954455-drjzm"
Jul 02 13:15:50.285513 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.285490    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 02 13:15:50.296640 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.296617    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-f5xn9\""
Jul 02 13:15:50.305421 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.305379    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 02 13:15:50.305935 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.305913    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 02 13:15:50.325597 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.325560    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-drjzm\" (UID: \"1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5\") " pod="openshift-monitoring/kube-state-metrics-845b954455-drjzm"
Jul 02 13:15:50.325801 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.325603    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-drjzm\" (UID: \"1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5\") " pod="openshift-monitoring/kube-state-metrics-845b954455-drjzm"
Jul 02 13:15:50.325801 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.325634    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-drjzm\" (UID: \"1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5\") " pod="openshift-monitoring/kube-state-metrics-845b954455-drjzm"
Jul 02 13:15:50.325801 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.325709    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42ttq\" (UniqueName: \"kubernetes.io/projected/1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5-kube-api-access-42ttq\") pod \"kube-state-metrics-845b954455-drjzm\" (UID: \"1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5\") " pod="openshift-monitoring/kube-state-metrics-845b954455-drjzm"
Jul 02 13:15:50.325801 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.325790    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-drjzm\" (UID: \"1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5\") " pod="openshift-monitoring/kube-state-metrics-845b954455-drjzm"
Jul 02 13:15:50.326030 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.325819    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5-metrics-client-ca\") pod \"kube-state-metrics-845b954455-drjzm\" (UID: \"1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5\") " pod="openshift-monitoring/kube-state-metrics-845b954455-drjzm"
Jul 02 13:15:50.326536 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.326507    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-n9cz7"]
Jul 02 13:15:50.329056 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.329033    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-drjzm"]
Jul 02 13:15:50.329205 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.329189    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-n9cz7"
Jul 02 13:15:50.342179 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.342151    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-6m5cb\""
Jul 02 13:15:50.342463 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.342450    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 02 13:15:50.351720 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.351701    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 02 13:15:50.372891 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.372858    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-n9cz7"]
Jul 02 13:15:50.390573 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.390533    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pjp9n"]
Jul 02 13:15:50.392796 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.392771    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pjp9n"
Jul 02 13:15:50.397034 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.397007    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 02 13:15:50.398480 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.398454    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 02 13:15:50.398480 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.398468    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f4hzn\""
Jul 02 13:15:50.398665 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.398501    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 02 13:15:50.417968 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.417938    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jlkds"
Jul 02 13:15:50.426908 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.426875    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z2qx\" (UniqueName: \"kubernetes.io/projected/60c1f844-3734-4f58-9ed9-f0b9f370b096-kube-api-access-5z2qx\") pod \"openshift-state-metrics-57f99c8b49-n9cz7\" (UID: \"60c1f844-3734-4f58-9ed9-f0b9f370b096\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-n9cz7"
Jul 02 13:15:50.427054 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.426910    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60c1f844-3734-4f58-9ed9-f0b9f370b096-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-n9cz7\" (UID: \"60c1f844-3734-4f58-9ed9-f0b9f370b096\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-n9cz7"
Jul 02 13:15:50.427054 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.426947    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-drjzm\" (UID: \"1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5\") " pod="openshift-monitoring/kube-state-metrics-845b954455-drjzm"
Jul 02 13:15:50.427054 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.426972    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-drjzm\" (UID: \"1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5\") " pod="openshift-monitoring/kube-state-metrics-845b954455-drjzm"
Jul 02 13:15:50.427054 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.427012    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a5e06a98-36dc-49b0-959c-d646b9802308-node-exporter-accelerators-collector-config\") pod \"node-exporter-pjp9n\" (UID: \"a5e06a98-36dc-49b0-959c-d646b9802308\") " pod="openshift-monitoring/node-exporter-pjp9n"
Jul 02 13:15:50.427054 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.427039    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42ttq\" (UniqueName: \"kubernetes.io/projected/1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5-kube-api-access-42ttq\") pod \"kube-state-metrics-845b954455-drjzm\" (UID: \"1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5\") " pod="openshift-monitoring/kube-state-metrics-845b954455-drjzm"
Jul 02 13:15:50.427228 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.427060    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a5e06a98-36dc-49b0-959c-d646b9802308-root\") pod \"node-exporter-pjp9n\" (UID: \"a5e06a98-36dc-49b0-959c-d646b9802308\") " pod="openshift-monitoring/node-exporter-pjp9n"
Jul 02 13:15:50.427228 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.427096    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-drjzm\" (UID: \"1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5\") " pod="openshift-monitoring/kube-state-metrics-845b954455-drjzm"
Jul 02 13:15:50.427228 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.427129    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a5e06a98-36dc-49b0-959c-d646b9802308-node-exporter-tls\") pod \"node-exporter-pjp9n\" (UID: \"a5e06a98-36dc-49b0-959c-d646b9802308\") " pod="openshift-monitoring/node-exporter-pjp9n"
Jul 02 13:15:50.427228 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.427153    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzcgz\" (UniqueName: \"kubernetes.io/projected/a5e06a98-36dc-49b0-959c-d646b9802308-kube-api-access-xzcgz\") pod \"node-exporter-pjp9n\" (UID: \"a5e06a98-36dc-49b0-959c-d646b9802308\") " pod="openshift-monitoring/node-exporter-pjp9n"
Jul 02 13:15:50.427425 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.427408    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a5e06a98-36dc-49b0-959c-d646b9802308-node-exporter-wtmp\") pod \"node-exporter-pjp9n\" (UID: \"a5e06a98-36dc-49b0-959c-d646b9802308\") " pod="openshift-monitoring/node-exporter-pjp9n"
Jul 02 13:15:50.427524 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.427503    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-drjzm\" (UID: \"1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5\") " pod="openshift-monitoring/kube-state-metrics-845b954455-drjzm"
Jul 02 13:15:50.427579 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.427501    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-drjzm\" (UID: \"1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5\") " pod="openshift-monitoring/kube-state-metrics-845b954455-drjzm"
Jul 02 13:15:50.427632 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:50.427584    2583 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 02 13:15:50.427632 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.427595    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5-metrics-client-ca\") pod \"kube-state-metrics-845b954455-drjzm\" (UID: \"1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5\") " pod="openshift-monitoring/kube-state-metrics-845b954455-drjzm"
Jul 02 13:15:50.427726 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:50.427648    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5-kube-state-metrics-tls podName:1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:50.927622857 +0000 UTC m=+62.155534022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5-kube-state-metrics-tls") pod "kube-state-metrics-845b954455-drjzm" (UID: "1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5") : secret "kube-state-metrics-tls" not found
Jul 02 13:15:50.427900 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.427862    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/60c1f844-3734-4f58-9ed9-f0b9f370b096-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-n9cz7\" (UID: \"60c1f844-3734-4f58-9ed9-f0b9f370b096\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-n9cz7"
Jul 02 13:15:50.427959 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.427936    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a5e06a98-36dc-49b0-959c-d646b9802308-node-exporter-textfile\") pod \"node-exporter-pjp9n\" (UID: \"a5e06a98-36dc-49b0-959c-d646b9802308\") " pod="openshift-monitoring/node-exporter-pjp9n"
Jul 02 13:15:50.428007 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.427976    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5e06a98-36dc-49b0-959c-d646b9802308-sys\") pod \"node-exporter-pjp9n\" (UID: \"a5e06a98-36dc-49b0-959c-d646b9802308\") " pod="openshift-monitoring/node-exporter-pjp9n"
Jul 02 13:15:50.428047 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.428021    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5e06a98-36dc-49b0-959c-d646b9802308-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pjp9n\" (UID: \"a5e06a98-36dc-49b0-959c-d646b9802308\") " pod="openshift-monitoring/node-exporter-pjp9n"
Jul 02 13:15:50.428093 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.428066    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-drjzm\" (UID: \"1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5\") " pod="openshift-monitoring/kube-state-metrics-845b954455-drjzm"
Jul 02 13:15:50.428127 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.428095    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60c1f844-3734-4f58-9ed9-f0b9f370b096-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-n9cz7\" (UID: \"60c1f844-3734-4f58-9ed9-f0b9f370b096\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-n9cz7"
Jul 02 13:15:50.428166 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.428124    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5e06a98-36dc-49b0-959c-d646b9802308-metrics-client-ca\") pod \"node-exporter-pjp9n\" (UID: \"a5e06a98-36dc-49b0-959c-d646b9802308\") " pod="openshift-monitoring/node-exporter-pjp9n"
Jul 02 13:15:50.428345 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.428327    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5-metrics-client-ca\") pod \"kube-state-metrics-845b954455-drjzm\" (UID: \"1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5\") " pod="openshift-monitoring/kube-state-metrics-845b954455-drjzm"
Jul 02 13:15:50.429444 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.429422    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-drjzm\" (UID: \"1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5\") " pod="openshift-monitoring/kube-state-metrics-845b954455-drjzm"
Jul 02 13:15:50.436811 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.436783    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:50.437673 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.437654    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42ttq\" (UniqueName: \"kubernetes.io/projected/1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5-kube-api-access-42ttq\") pod \"kube-state-metrics-845b954455-drjzm\" (UID: \"1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5\") " pod="openshift-monitoring/kube-state-metrics-845b954455-drjzm"
Jul 02 13:15:50.512654 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.512563    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n5cmj" event={"ID":"3810cfce-45d5-47b2-950c-a17f109aa7ac","Type":"ContainerStarted","Data":"15a4c86067fc8c0e48ce1ed86416c1a8309aa75d26040b195df8fd4671e80f60"}
Jul 02 13:15:50.528554 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.528521    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/60c1f844-3734-4f58-9ed9-f0b9f370b096-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-n9cz7\" (UID: \"60c1f844-3734-4f58-9ed9-f0b9f370b096\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-n9cz7"
Jul 02 13:15:50.528554 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.528556    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a5e06a98-36dc-49b0-959c-d646b9802308-node-exporter-textfile\") pod \"node-exporter-pjp9n\" (UID: \"a5e06a98-36dc-49b0-959c-d646b9802308\") " pod="openshift-monitoring/node-exporter-pjp9n"
Jul 02 13:15:50.528764 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.528575    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5e06a98-36dc-49b0-959c-d646b9802308-sys\") pod \"node-exporter-pjp9n\" (UID: \"a5e06a98-36dc-49b0-959c-d646b9802308\") " pod="openshift-monitoring/node-exporter-pjp9n"
Jul 02 13:15:50.528764 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.528636    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5e06a98-36dc-49b0-959c-d646b9802308-sys\") pod \"node-exporter-pjp9n\" (UID: \"a5e06a98-36dc-49b0-959c-d646b9802308\") " pod="openshift-monitoring/node-exporter-pjp9n"
Jul 02 13:15:50.528764 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.528706    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5e06a98-36dc-49b0-959c-d646b9802308-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pjp9n\" (UID: \"a5e06a98-36dc-49b0-959c-d646b9802308\") " pod="openshift-monitoring/node-exporter-pjp9n"
Jul 02 13:15:50.528898 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.528771    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60c1f844-3734-4f58-9ed9-f0b9f370b096-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-n9cz7\" (UID: \"60c1f844-3734-4f58-9ed9-f0b9f370b096\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-n9cz7"
Jul 02 13:15:50.528898 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.528798    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5e06a98-36dc-49b0-959c-d646b9802308-metrics-client-ca\") pod \"node-exporter-pjp9n\" (UID: \"a5e06a98-36dc-49b0-959c-d646b9802308\") " pod="openshift-monitoring/node-exporter-pjp9n"
Jul 02 13:15:50.528898 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.528832    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z2qx\" (UniqueName: \"kubernetes.io/projected/60c1f844-3734-4f58-9ed9-f0b9f370b096-kube-api-access-5z2qx\") pod \"openshift-state-metrics-57f99c8b49-n9cz7\" (UID: \"60c1f844-3734-4f58-9ed9-f0b9f370b096\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-n9cz7"
Jul 02 13:15:50.528898 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.528857    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60c1f844-3734-4f58-9ed9-f0b9f370b096-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-n9cz7\" (UID: \"60c1f844-3734-4f58-9ed9-f0b9f370b096\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-n9cz7"
Jul 02 13:15:50.529075 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.528911    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a5e06a98-36dc-49b0-959c-d646b9802308-node-exporter-accelerators-collector-config\") pod \"node-exporter-pjp9n\" (UID: \"a5e06a98-36dc-49b0-959c-d646b9802308\") " pod="openshift-monitoring/node-exporter-pjp9n"
Jul 02 13:15:50.529075 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.528938    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a5e06a98-36dc-49b0-959c-d646b9802308-root\") pod \"node-exporter-pjp9n\" (UID: \"a5e06a98-36dc-49b0-959c-d646b9802308\") " pod="openshift-monitoring/node-exporter-pjp9n"
Jul 02 13:15:50.529075 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.528966    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a5e06a98-36dc-49b0-959c-d646b9802308-node-exporter-textfile\") pod \"node-exporter-pjp9n\" (UID: \"a5e06a98-36dc-49b0-959c-d646b9802308\") " pod="openshift-monitoring/node-exporter-pjp9n"
Jul 02 13:15:50.529075 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.528971    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a5e06a98-36dc-49b0-959c-d646b9802308-node-exporter-tls\") pod \"node-exporter-pjp9n\" (UID: \"a5e06a98-36dc-49b0-959c-d646b9802308\") " pod="openshift-monitoring/node-exporter-pjp9n"
Jul 02 13:15:50.529075 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.529016    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzcgz\" (UniqueName: \"kubernetes.io/projected/a5e06a98-36dc-49b0-959c-d646b9802308-kube-api-access-xzcgz\") pod \"node-exporter-pjp9n\" (UID: \"a5e06a98-36dc-49b0-959c-d646b9802308\") " pod="openshift-monitoring/node-exporter-pjp9n"
Jul 02 13:15:50.529075 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:50.529048    2583 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 02 13:15:50.529346 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:50.529104    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5e06a98-36dc-49b0-959c-d646b9802308-node-exporter-tls podName:a5e06a98-36dc-49b0-959c-d646b9802308 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:51.029085651 +0000 UTC m=+62.256996807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/a5e06a98-36dc-49b0-959c-d646b9802308-node-exporter-tls") pod "node-exporter-pjp9n" (UID: "a5e06a98-36dc-49b0-959c-d646b9802308") : secret "node-exporter-tls" not found
Jul 02 13:15:50.529346 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.529166    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a5e06a98-36dc-49b0-959c-d646b9802308-node-exporter-wtmp\") pod \"node-exporter-pjp9n\" (UID: \"a5e06a98-36dc-49b0-959c-d646b9802308\") " pod="openshift-monitoring/node-exporter-pjp9n"
Jul 02 13:15:50.529346 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.529049    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a5e06a98-36dc-49b0-959c-d646b9802308-node-exporter-wtmp\") pod \"node-exporter-pjp9n\" (UID: \"a5e06a98-36dc-49b0-959c-d646b9802308\") " pod="openshift-monitoring/node-exporter-pjp9n"
Jul 02 13:15:50.529346 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.529205    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a5e06a98-36dc-49b0-959c-d646b9802308-root\") pod \"node-exporter-pjp9n\" (UID: \"a5e06a98-36dc-49b0-959c-d646b9802308\") " pod="openshift-monitoring/node-exporter-pjp9n"
Jul 02 13:15:50.529346 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.529320    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5e06a98-36dc-49b0-959c-d646b9802308-metrics-client-ca\") pod \"node-exporter-pjp9n\" (UID: \"a5e06a98-36dc-49b0-959c-d646b9802308\") " pod="openshift-monitoring/node-exporter-pjp9n"
Jul 02 13:15:50.529667 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.529647    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60c1f844-3734-4f58-9ed9-f0b9f370b096-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-n9cz7\" (UID: \"60c1f844-3734-4f58-9ed9-f0b9f370b096\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-n9cz7"
Jul 02 13:15:50.529718 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.529704    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a5e06a98-36dc-49b0-959c-d646b9802308-node-exporter-accelerators-collector-config\") pod \"node-exporter-pjp9n\" (UID: \"a5e06a98-36dc-49b0-959c-d646b9802308\") " pod="openshift-monitoring/node-exporter-pjp9n"
Jul 02 13:15:50.531104 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.531084    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/60c1f844-3734-4f58-9ed9-f0b9f370b096-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-n9cz7\" (UID: \"60c1f844-3734-4f58-9ed9-f0b9f370b096\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-n9cz7"
Jul 02 13:15:50.531795 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.531775    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5e06a98-36dc-49b0-959c-d646b9802308-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pjp9n\" (UID: \"a5e06a98-36dc-49b0-959c-d646b9802308\") " pod="openshift-monitoring/node-exporter-pjp9n"
Jul 02 13:15:50.531867 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.531840    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60c1f844-3734-4f58-9ed9-f0b9f370b096-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-n9cz7\" (UID: \"60c1f844-3734-4f58-9ed9-f0b9f370b096\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-n9cz7"
Jul 02 13:15:50.554757 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.554725    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z2qx\" (UniqueName: \"kubernetes.io/projected/60c1f844-3734-4f58-9ed9-f0b9f370b096-kube-api-access-5z2qx\") pod \"openshift-state-metrics-57f99c8b49-n9cz7\" (UID: \"60c1f844-3734-4f58-9ed9-f0b9f370b096\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-n9cz7"
Jul 02 13:15:50.568723 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.568692    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzcgz\" (UniqueName: \"kubernetes.io/projected/a5e06a98-36dc-49b0-959c-d646b9802308-kube-api-access-xzcgz\") pod \"node-exporter-pjp9n\" (UID: \"a5e06a98-36dc-49b0-959c-d646b9802308\") " pod="openshift-monitoring/node-exporter-pjp9n"
Jul 02 13:15:50.626186 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.626152    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:50.638358 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.638323    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-n9cz7"
Jul 02 13:15:50.675575 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.675508    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-n5cmj" podStartSLOduration=1.6587935059999999 podStartE2EDuration="13.675485003s" totalImagesPullingTime="12.016691497s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:15:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:50.578344664 +0000 UTC m=+61.806255838" watchObservedRunningTime="2026-07-02 13:15:50.675485003 +0000 UTC m=+61.903396176"
Jul 02 13:15:50.676620 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.676596    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5499b6fb8f-mxb5n"]
Jul 02 13:15:50.683753 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.683726    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5499b6fb8f-mxb5n"
Jul 02 13:15:50.691334 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.691308    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 02 13:15:50.691608 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.691590    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-gcgcw\""
Jul 02 13:15:50.695641 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.695620    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 02 13:15:50.709464 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.709425    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5499b6fb8f-mxb5n"]
Jul 02 13:15:50.731302 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.731267    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqr78\" (UniqueName: \"kubernetes.io/projected/79b6d559-f1fe-45a4-9402-5c4008c2108b-kube-api-access-mqr78\") pod \"downloads-5499b6fb8f-mxb5n\" (UID: \"79b6d559-f1fe-45a4-9402-5c4008c2108b\") " pod="openshift-console/downloads-5499b6fb8f-mxb5n"
Jul 02 13:15:50.793363 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.793283    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-n9cz7"]
Jul 02 13:15:50.796622 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:15:50.796596    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c1f844_3734_4f58_9ed9_f0b9f370b096.slice/crio-58d31771f00d2f403ecefe8d5c2fd4e5418dc7af7fc10f8b730966056c1c1d81 WatchSource:0}: Error finding container 58d31771f00d2f403ecefe8d5c2fd4e5418dc7af7fc10f8b730966056c1c1d81: Status 404 returned error can't find the container with id 58d31771f00d2f403ecefe8d5c2fd4e5418dc7af7fc10f8b730966056c1c1d81
Jul 02 13:15:50.810177 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.810145    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:50.832186 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.832161    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqr78\" (UniqueName: \"kubernetes.io/projected/79b6d559-f1fe-45a4-9402-5c4008c2108b-kube-api-access-mqr78\") pod \"downloads-5499b6fb8f-mxb5n\" (UID: \"79b6d559-f1fe-45a4-9402-5c4008c2108b\") " pod="openshift-console/downloads-5499b6fb8f-mxb5n"
Jul 02 13:15:50.844986 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.844964    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqr78\" (UniqueName: \"kubernetes.io/projected/79b6d559-f1fe-45a4-9402-5c4008c2108b-kube-api-access-mqr78\") pod \"downloads-5499b6fb8f-mxb5n\" (UID: \"79b6d559-f1fe-45a4-9402-5c4008c2108b\") " pod="openshift-console/downloads-5499b6fb8f-mxb5n"
Jul 02 13:15:50.923622 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.923593    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:50.932957 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.932925    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-drjzm\" (UID: \"1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5\") " pod="openshift-monitoring/kube-state-metrics-845b954455-drjzm"
Jul 02 13:15:50.935370 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.935346    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-drjzm\" (UID: \"1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5\") " pod="openshift-monitoring/kube-state-metrics-845b954455-drjzm"
Jul 02 13:15:50.996211 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:50.996180    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5499b6fb8f-mxb5n"
Jul 02 13:15:51.024921 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.024894    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:51.035174 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.035139    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a5e06a98-36dc-49b0-959c-d646b9802308-node-exporter-tls\") pod \"node-exporter-pjp9n\" (UID: \"a5e06a98-36dc-49b0-959c-d646b9802308\") " pod="openshift-monitoring/node-exporter-pjp9n"
Jul 02 13:15:51.037969 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.037934    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a5e06a98-36dc-49b0-959c-d646b9802308-node-exporter-tls\") pod \"node-exporter-pjp9n\" (UID: \"a5e06a98-36dc-49b0-959c-d646b9802308\") " pod="openshift-monitoring/node-exporter-pjp9n"
Jul 02 13:15:51.132146 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.132108    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5499b6fb8f-mxb5n"]
Jul 02 13:15:51.136069 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:15:51.136043    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b6d559_f1fe_45a4_9402_5c4008c2108b.slice/crio-798bd2a724a7840d104992e8b19c94283846fe7623f483554a205729fda16e7f WatchSource:0}: Error finding container 798bd2a724a7840d104992e8b19c94283846fe7623f483554a205729fda16e7f: Status 404 returned error can't find the container with id 798bd2a724a7840d104992e8b19c94283846fe7623f483554a205729fda16e7f
Jul 02 13:15:51.138917 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.138851    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:51.191407 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.191354    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-845b954455-drjzm"
Jul 02 13:15:51.205929 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.205904    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 13:15:51.208515 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.208499    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.212047 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.211609    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 02 13:15:51.212047 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.211849    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 02 13:15:51.212047 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.211976    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 02 13:15:51.212047 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.211993    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-qqtmd\""
Jul 02 13:15:51.212047 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.212008    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 02 13:15:51.212414 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.212284    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 02 13:15:51.212543 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.212527    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 02 13:15:51.212615 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.212532    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 02 13:15:51.212615 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.212559    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 02 13:15:51.213217 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.213200    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 02 13:15:51.237227 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.237103    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.237528 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.237498    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.237640 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.237562    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx9q8\" (UniqueName: \"kubernetes.io/projected/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-kube-api-access-qx9q8\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.237640 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.237603    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-config-volume\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.237770 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.237656    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.237770 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.237694    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.237770 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.237758    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.237890 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.237779    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.237890 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.237802    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-config-out\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.237890 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.237823    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.237890 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.237860    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.238041 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.237890    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.238041 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.237926    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-web-config\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.239231 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.239202    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 13:15:51.242224 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.242199    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:51.301804 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.301730    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pjp9n"
Jul 02 13:15:51.311181 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:15:51.311145    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5e06a98_36dc_49b0_959c_d646b9802308.slice/crio-2da2b5aaa98f9fed8233cf67995f02b7985e5580d1f012a08152cdcc5432f559 WatchSource:0}: Error finding container 2da2b5aaa98f9fed8233cf67995f02b7985e5580d1f012a08152cdcc5432f559: Status 404 returned error can't find the container with id 2da2b5aaa98f9fed8233cf67995f02b7985e5580d1f012a08152cdcc5432f559
Jul 02 13:15:51.338461 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.338420    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.339319 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.338511    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx9q8\" (UniqueName: \"kubernetes.io/projected/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-kube-api-access-qx9q8\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.339319 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.338537    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-config-volume\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.339319 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.338557    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.339319 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.338580    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.339319 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.338620    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.339319 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.338644    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.339319 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.338675    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-config-out\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.339319 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.338704    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.339319 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.338752    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.339319 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.338780    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.339319 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.338814    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-web-config\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.339319 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.338851    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.339962 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.339701    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.340101 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.340075    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-drjzm"]
Jul 02 13:15:51.340209 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:51.338673    2583 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 02 13:15:51.340209 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.340172    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.340209 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:51.340207    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-secret-alertmanager-main-tls podName:c2cc0d64-e3d1-429e-8eda-f07d72ceab95 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:51.84018799 +0000 UTC m=+63.068099159 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "c2cc0d64-e3d1-429e-8eda-f07d72ceab95") : secret "alertmanager-main-tls" not found
Jul 02 13:15:51.340444 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:15:51.340302    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-alertmanager-trusted-ca-bundle podName:c2cc0d64-e3d1-429e-8eda-f07d72ceab95 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:51.840290372 +0000 UTC m=+63.068201544 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "c2cc0d64-e3d1-429e-8eda-f07d72ceab95") : configmap references non-existent config key: ca-bundle.crt
Jul 02 13:15:51.341825 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.341797    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.342144 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.342120    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.342227 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.342183    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.342704 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.342680    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-config-volume\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.342834 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.342764    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-web-config\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.343125 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.343088    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.343125 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.343092    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-config-out\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.344237 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.344215    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.344847 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.344808    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:51.345161 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:15:51.345143    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f57e6e5_ce15_4e7f_9fcd_12f0de2546c5.slice/crio-480200bf5534658e52f17d644ad33c6e0da50d7e82debaf0b9b7af7935f5ef40 WatchSource:0}: Error finding container 480200bf5534658e52f17d644ad33c6e0da50d7e82debaf0b9b7af7935f5ef40: Status 404 returned error can't find the container with id 480200bf5534658e52f17d644ad33c6e0da50d7e82debaf0b9b7af7935f5ef40
Jul 02 13:15:51.349720 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.349701    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx9q8\" (UniqueName: \"kubernetes.io/projected/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-kube-api-access-qx9q8\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.460800 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.460729    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:51.521471 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.521417    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5499b6fb8f-mxb5n" event={"ID":"79b6d559-f1fe-45a4-9402-5c4008c2108b","Type":"ContainerStarted","Data":"798bd2a724a7840d104992e8b19c94283846fe7623f483554a205729fda16e7f"}
Jul 02 13:15:51.522758 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.522723    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pjp9n" event={"ID":"a5e06a98-36dc-49b0-959c-d646b9802308","Type":"ContainerStarted","Data":"2da2b5aaa98f9fed8233cf67995f02b7985e5580d1f012a08152cdcc5432f559"}
Jul 02 13:15:51.524792 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.524766    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-n9cz7" event={"ID":"60c1f844-3734-4f58-9ed9-f0b9f370b096","Type":"ContainerStarted","Data":"e936365dee47298ed1f32f747b4c50d1ec83b6deb56981c2a20a861de78d34f0"}
Jul 02 13:15:51.524935 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.524822    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-n9cz7" event={"ID":"60c1f844-3734-4f58-9ed9-f0b9f370b096","Type":"ContainerStarted","Data":"746e2f85d3f150608388e8a45a474526d161b9a60157fbcfb186289f58592b2d"}
Jul 02 13:15:51.524935 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.524853    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-n9cz7" event={"ID":"60c1f844-3734-4f58-9ed9-f0b9f370b096","Type":"ContainerStarted","Data":"58d31771f00d2f403ecefe8d5c2fd4e5418dc7af7fc10f8b730966056c1c1d81"}
Jul 02 13:15:51.526027 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.525994    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-drjzm" event={"ID":"1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5","Type":"ContainerStarted","Data":"480200bf5534658e52f17d644ad33c6e0da50d7e82debaf0b9b7af7935f5ef40"}
Jul 02 13:15:51.667254 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.667210    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:51.767450 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.767414    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:51.843318 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.843285    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.843493 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.843368    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.844264 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.844242    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:51.846438 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:51.846418    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:52.073675 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.073596    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x"]
Jul 02 13:15:52.076563 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.076540    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x"
Jul 02 13:15:52.082658 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.082357    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 02 13:15:52.082658 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.082357    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 02 13:15:52.082658 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.082674    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 02 13:15:52.082931 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.082691    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 02 13:15:52.082931 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.082786    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-bp78b15s3dpup\""
Jul 02 13:15:52.082931 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.082920    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-nr7rk\""
Jul 02 13:15:52.085827 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.085808    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 02 13:15:52.109229 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.109194    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x"]
Jul 02 13:15:52.118381 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.118354    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:15:52.146851 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.146816    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57f4dddd4d-p6t8x\" (UID: \"5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3\") " pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x"
Jul 02 13:15:52.147019 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.146936    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57f4dddd4d-p6t8x\" (UID: \"5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3\") " pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x"
Jul 02 13:15:52.147019 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.146970    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3-secret-grpc-tls\") pod \"thanos-querier-57f4dddd4d-p6t8x\" (UID: \"5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3\") " pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x"
Jul 02 13:15:52.147123 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.147022    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr22t\" (UniqueName: \"kubernetes.io/projected/5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3-kube-api-access-mr22t\") pod \"thanos-querier-57f4dddd4d-p6t8x\" (UID: \"5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3\") " pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x"
Jul 02 13:15:52.147123 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.147108    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57f4dddd4d-p6t8x\" (UID: \"5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3\") " pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x"
Jul 02 13:15:52.147199 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.147145    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57f4dddd4d-p6t8x\" (UID: \"5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3\") " pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x"
Jul 02 13:15:52.147199 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.147178    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3-secret-thanos-querier-tls\") pod \"thanos-querier-57f4dddd4d-p6t8x\" (UID: \"5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3\") " pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x"
Jul 02 13:15:52.147279 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.147260    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3-metrics-client-ca\") pod \"thanos-querier-57f4dddd4d-p6t8x\" (UID: \"5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3\") " pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x"
Jul 02 13:15:52.188351 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.188316    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:52.248447 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.248414    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57f4dddd4d-p6t8x\" (UID: \"5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3\") " pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x"
Jul 02 13:15:52.248621 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.248456    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57f4dddd4d-p6t8x\" (UID: \"5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3\") " pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x"
Jul 02 13:15:52.248684 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.248609    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3-secret-thanos-querier-tls\") pod \"thanos-querier-57f4dddd4d-p6t8x\" (UID: \"5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3\") " pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x"
Jul 02 13:15:52.248684 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.248664    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3-metrics-client-ca\") pod \"thanos-querier-57f4dddd4d-p6t8x\" (UID: \"5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3\") " pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x"
Jul 02 13:15:52.248789 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.248715    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57f4dddd4d-p6t8x\" (UID: \"5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3\") " pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x"
Jul 02 13:15:52.248839 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.248807    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57f4dddd4d-p6t8x\" (UID: \"5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3\") " pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x"
Jul 02 13:15:52.248890 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.248838    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3-secret-grpc-tls\") pod \"thanos-querier-57f4dddd4d-p6t8x\" (UID: \"5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3\") " pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x"
Jul 02 13:15:52.248939 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.248894    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr22t\" (UniqueName: \"kubernetes.io/projected/5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3-kube-api-access-mr22t\") pod \"thanos-querier-57f4dddd4d-p6t8x\" (UID: \"5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3\") " pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x"
Jul 02 13:15:52.250094 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.250012    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3-metrics-client-ca\") pod \"thanos-querier-57f4dddd4d-p6t8x\" (UID: \"5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3\") " pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x"
Jul 02 13:15:52.251493 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.251444    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57f4dddd4d-p6t8x\" (UID: \"5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3\") " pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x"
Jul 02 13:15:52.251612 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.251574    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3-secret-thanos-querier-tls\") pod \"thanos-querier-57f4dddd4d-p6t8x\" (UID: \"5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3\") " pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x"
Jul 02 13:15:52.252288 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.252244    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57f4dddd4d-p6t8x\" (UID: \"5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3\") " pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x"
Jul 02 13:15:52.253025 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.252994    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57f4dddd4d-p6t8x\" (UID: \"5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3\") " pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x"
Jul 02 13:15:52.253225 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.253175    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3-secret-grpc-tls\") pod \"thanos-querier-57f4dddd4d-p6t8x\" (UID: \"5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3\") " pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x"
Jul 02 13:15:52.253777 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.253754    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57f4dddd4d-p6t8x\" (UID: \"5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3\") " pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x"
Jul 02 13:15:52.261291 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.261235    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr22t\" (UniqueName: \"kubernetes.io/projected/5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3-kube-api-access-mr22t\") pod \"thanos-querier-57f4dddd4d-p6t8x\" (UID: \"5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3\") " pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x"
Jul 02 13:15:52.288950 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.288908    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 13:15:52.289126 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.288957    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:52.293506 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:15:52.293475    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2cc0d64_e3d1_429e_8eda_f07d72ceab95.slice/crio-1f91c0a76f0abf1c957e9f9c4b728ebe2ee966c6ab2f08912af49229c6d24433 WatchSource:0}: Error finding container 1f91c0a76f0abf1c957e9f9c4b728ebe2ee966c6ab2f08912af49229c6d24433: Status 404 returned error can't find the container with id 1f91c0a76f0abf1c957e9f9c4b728ebe2ee966c6ab2f08912af49229c6d24433
Jul 02 13:15:52.389184 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.389149    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x"
Jul 02 13:15:52.530761 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.530721    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c2cc0d64-e3d1-429e-8eda-f07d72ceab95","Type":"ContainerStarted","Data":"1f91c0a76f0abf1c957e9f9c4b728ebe2ee966c6ab2f08912af49229c6d24433"}
Jul 02 13:15:52.533104 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.533070    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-n9cz7" event={"ID":"60c1f844-3734-4f58-9ed9-f0b9f370b096","Type":"ContainerStarted","Data":"2f64c82dc58dd8a9acc02b4d666859fa211f0051102be81bf4967719b334c4da"}
Jul 02 13:15:52.553380 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.553329    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-n9cz7" podStartSLOduration=1.613151501 podStartE2EDuration="2.553314236s" totalImagesPullingTime="940.162735ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:52.551944158 +0000 UTC m=+63.779855331" watchObservedRunningTime="2026-07-02 13:15:52.553314236 +0000 UTC m=+63.781225410"
Jul 02 13:15:52.747856 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.747780    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:52.810188 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:52.810152    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x"]
Jul 02 13:15:52.817430 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:15:52.817370    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c97d97e_b6a7_4f3f_80b8_e2dbd6857ba3.slice/crio-126c6216d9f7006310c913228f893f7712168db64354012438c8bff60c9b28f7 WatchSource:0}: Error finding container 126c6216d9f7006310c913228f893f7712168db64354012438c8bff60c9b28f7: Status 404 returned error can't find the container with id 126c6216d9f7006310c913228f893f7712168db64354012438c8bff60c9b28f7
Jul 02 13:15:53.064923 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:53.064829    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:53.166067 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:53.166025    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:53.483009 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:53.482971    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:53.539045 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:53.539005    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-drjzm" event={"ID":"1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5","Type":"ContainerStarted","Data":"ef79a29fadc5883a7d4bf85bd349d68e01c6c286dc01380332255e2d6d85f1fb"}
Jul 02 13:15:53.539238 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:53.539072    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-drjzm" event={"ID":"1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5","Type":"ContainerStarted","Data":"aa7fbd202754891aa01ffd7b16f38114eebdcc2ffe620700999300d2f4b3038c"}
Jul 02 13:15:53.539238 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:53.539099    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-drjzm" event={"ID":"1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5","Type":"ContainerStarted","Data":"76e75266a92adf20d5f5951c840f654fca7088cccd096b3ce6bdc739b003bb68"}
Jul 02 13:15:53.540233 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:53.540200    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x" event={"ID":"5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3","Type":"ContainerStarted","Data":"126c6216d9f7006310c913228f893f7712168db64354012438c8bff60c9b28f7"}
Jul 02 13:15:53.541661 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:53.541624    2583 generic.go:356] "Generic (PLEG): container finished" podID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" containerID="2bade4f1d8a8dd313d3ca7dfc3553c3832b8c32c695c43aaf864cebfb205bba4" exitCode=0
Jul 02 13:15:53.541777 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:53.541715    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c2cc0d64-e3d1-429e-8eda-f07d72ceab95","Type":"ContainerDied","Data":"2bade4f1d8a8dd313d3ca7dfc3553c3832b8c32c695c43aaf864cebfb205bba4"}
Jul 02 13:15:53.567083 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:53.566977    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-845b954455-drjzm" podStartSLOduration=2.241106125 podStartE2EDuration="3.566961602s" totalImagesPullingTime="1.325855477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:53.566117068 +0000 UTC m=+64.794028253" watchObservedRunningTime="2026-07-02 13:15:53.566961602 +0000 UTC m=+64.794872776"
Jul 02 13:15:53.571294 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:53.571265    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:54.547298 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:54.547258    2583 generic.go:356] "Generic (PLEG): container finished" podID="a5e06a98-36dc-49b0-959c-d646b9802308" containerID="02336af0cecaeca81d4a9da1952e5f304006284f70d2c84ac756a3c8ec029417" exitCode=0
Jul 02 13:15:54.547831 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:54.547348    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pjp9n" event={"ID":"a5e06a98-36dc-49b0-959c-d646b9802308","Type":"ContainerDied","Data":"02336af0cecaeca81d4a9da1952e5f304006284f70d2c84ac756a3c8ec029417"}
Jul 02 13:15:54.722236 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:54.722195    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:54.737164 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:54.736161    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-cd8656b8-fq7jd"]
Jul 02 13:15:54.739372 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:54.738675    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-cd8656b8-fq7jd"
Jul 02 13:15:54.742969 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:54.742945    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-p798t\""
Jul 02 13:15:54.743257 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:54.743228    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 02 13:15:54.764689 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:54.764652    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-cd8656b8-fq7jd"]
Jul 02 13:15:54.774968 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:54.774648    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8dn2\" (UniqueName: \"kubernetes.io/projected/6e4c183e-20bc-408d-b975-bc6a9aafe76c-kube-api-access-t8dn2\") pod \"monitoring-plugin-cd8656b8-fq7jd\" (UID: \"6e4c183e-20bc-408d-b975-bc6a9aafe76c\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-fq7jd"
Jul 02 13:15:54.774968 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:54.774735    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6e4c183e-20bc-408d-b975-bc6a9aafe76c-monitoring-plugin-cert\") pod \"monitoring-plugin-cd8656b8-fq7jd\" (UID: \"6e4c183e-20bc-408d-b975-bc6a9aafe76c\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-fq7jd"
Jul 02 13:15:54.875910 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:54.875462    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6e4c183e-20bc-408d-b975-bc6a9aafe76c-monitoring-plugin-cert\") pod \"monitoring-plugin-cd8656b8-fq7jd\" (UID: \"6e4c183e-20bc-408d-b975-bc6a9aafe76c\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-fq7jd"
Jul 02 13:15:54.875910 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:54.875591    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8dn2\" (UniqueName: \"kubernetes.io/projected/6e4c183e-20bc-408d-b975-bc6a9aafe76c-kube-api-access-t8dn2\") pod \"monitoring-plugin-cd8656b8-fq7jd\" (UID: \"6e4c183e-20bc-408d-b975-bc6a9aafe76c\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-fq7jd"
Jul 02 13:15:54.878951 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:54.878920    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6e4c183e-20bc-408d-b975-bc6a9aafe76c-monitoring-plugin-cert\") pod \"monitoring-plugin-cd8656b8-fq7jd\" (UID: \"6e4c183e-20bc-408d-b975-bc6a9aafe76c\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-fq7jd"
Jul 02 13:15:54.888351 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:54.888318    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8dn2\" (UniqueName: \"kubernetes.io/projected/6e4c183e-20bc-408d-b975-bc6a9aafe76c-kube-api-access-t8dn2\") pod \"monitoring-plugin-cd8656b8-fq7jd\" (UID: \"6e4c183e-20bc-408d-b975-bc6a9aafe76c\") " pod="openshift-monitoring/monitoring-plugin-cd8656b8-fq7jd"
Jul 02 13:15:54.976222 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:54.975558    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7557bdc5fb-gmwtd"]
Jul 02 13:15:54.976381 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:54.976266    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cae18b4-8f5b-4ff7-a73f-fc668af0a663-metrics-certs\") pod \"network-metrics-daemon-xzl6q\" (UID: \"0cae18b4-8f5b-4ff7-a73f-fc668af0a663\") " pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:15:54.976381 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:54.976338    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bfa11a9f-ca92-4214-8e84-0b9c48a8337c-original-pull-secret\") pod \"global-pull-secret-syncer-9fpt5\" (UID: \"bfa11a9f-ca92-4214-8e84-0b9c48a8337c\") " pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:15:54.978502 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:54.978454    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7557bdc5fb-gmwtd"
Jul 02 13:15:54.979582 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:54.979560    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 13:15:54.979696 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:54.979608    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 13:15:54.981016 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:54.980996    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 02 13:15:54.981543 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:54.981336    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-ljmh2\""
Jul 02 13:15:54.981543 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:54.981351    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 02 13:15:54.981543 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:54.981351    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 02 13:15:54.981764 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:54.981733    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-373spd5l4ofhc\""
Jul 02 13:15:54.981980 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:54.981962    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 02 13:15:54.992852 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:54.990693    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bfa11a9f-ca92-4214-8e84-0b9c48a8337c-original-pull-secret\") pod \"global-pull-secret-syncer-9fpt5\" (UID: \"bfa11a9f-ca92-4214-8e84-0b9c48a8337c\") " pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:15:54.992852 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:54.990734    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cae18b4-8f5b-4ff7-a73f-fc668af0a663-metrics-certs\") pod \"network-metrics-daemon-xzl6q\" (UID: \"0cae18b4-8f5b-4ff7-a73f-fc668af0a663\") " pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:15:54.992852 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:54.992630    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7557bdc5fb-gmwtd"]
Jul 02 13:15:55.036633 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.036598    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9fpt5"
Jul 02 13:15:55.051079 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.051045    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-cd8656b8-fq7jd"
Jul 02 13:15:55.077044 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.077014    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmvc4\" (UniqueName: \"kubernetes.io/projected/886e2635-8f28-407b-99e6-cb8a3f869117-kube-api-access-fmvc4\") pod \"metrics-server-7557bdc5fb-gmwtd\" (UID: \"886e2635-8f28-407b-99e6-cb8a3f869117\") " pod="openshift-monitoring/metrics-server-7557bdc5fb-gmwtd"
Jul 02 13:15:55.077182 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.077052    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/886e2635-8f28-407b-99e6-cb8a3f869117-secret-metrics-server-tls\") pod \"metrics-server-7557bdc5fb-gmwtd\" (UID: \"886e2635-8f28-407b-99e6-cb8a3f869117\") " pod="openshift-monitoring/metrics-server-7557bdc5fb-gmwtd"
Jul 02 13:15:55.077182 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.077081    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/886e2635-8f28-407b-99e6-cb8a3f869117-audit-log\") pod \"metrics-server-7557bdc5fb-gmwtd\" (UID: \"886e2635-8f28-407b-99e6-cb8a3f869117\") " pod="openshift-monitoring/metrics-server-7557bdc5fb-gmwtd"
Jul 02 13:15:55.077182 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.077102    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/886e2635-8f28-407b-99e6-cb8a3f869117-metrics-server-audit-profiles\") pod \"metrics-server-7557bdc5fb-gmwtd\" (UID: \"886e2635-8f28-407b-99e6-cb8a3f869117\") " pod="openshift-monitoring/metrics-server-7557bdc5fb-gmwtd"
Jul 02 13:15:55.077182 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.077128    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/886e2635-8f28-407b-99e6-cb8a3f869117-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7557bdc5fb-gmwtd\" (UID: \"886e2635-8f28-407b-99e6-cb8a3f869117\") " pod="openshift-monitoring/metrics-server-7557bdc5fb-gmwtd"
Jul 02 13:15:55.077182 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.077147    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/886e2635-8f28-407b-99e6-cb8a3f869117-client-ca-bundle\") pod \"metrics-server-7557bdc5fb-gmwtd\" (UID: \"886e2635-8f28-407b-99e6-cb8a3f869117\") " pod="openshift-monitoring/metrics-server-7557bdc5fb-gmwtd"
Jul 02 13:15:55.077421 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.077293    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/886e2635-8f28-407b-99e6-cb8a3f869117-secret-metrics-server-client-certs\") pod \"metrics-server-7557bdc5fb-gmwtd\" (UID: \"886e2635-8f28-407b-99e6-cb8a3f869117\") " pod="openshift-monitoring/metrics-server-7557bdc5fb-gmwtd"
Jul 02 13:15:55.178418 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.178310    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/886e2635-8f28-407b-99e6-cb8a3f869117-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7557bdc5fb-gmwtd\" (UID: \"886e2635-8f28-407b-99e6-cb8a3f869117\") " pod="openshift-monitoring/metrics-server-7557bdc5fb-gmwtd"
Jul 02 13:15:55.178418 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.178358    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/886e2635-8f28-407b-99e6-cb8a3f869117-client-ca-bundle\") pod \"metrics-server-7557bdc5fb-gmwtd\" (UID: \"886e2635-8f28-407b-99e6-cb8a3f869117\") " pod="openshift-monitoring/metrics-server-7557bdc5fb-gmwtd"
Jul 02 13:15:55.178654 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.178441    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/886e2635-8f28-407b-99e6-cb8a3f869117-secret-metrics-server-client-certs\") pod \"metrics-server-7557bdc5fb-gmwtd\" (UID: \"886e2635-8f28-407b-99e6-cb8a3f869117\") " pod="openshift-monitoring/metrics-server-7557bdc5fb-gmwtd"
Jul 02 13:15:55.178654 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.178489    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2r82\" (UniqueName: \"kubernetes.io/projected/159f78cc-1fe0-40ca-a0cd-92254294389c-kube-api-access-s2r82\") pod \"network-check-target-tx24z\" (UID: \"159f78cc-1fe0-40ca-a0cd-92254294389c\") " pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:15:55.178654 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.178514    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmvc4\" (UniqueName: \"kubernetes.io/projected/886e2635-8f28-407b-99e6-cb8a3f869117-kube-api-access-fmvc4\") pod \"metrics-server-7557bdc5fb-gmwtd\" (UID: \"886e2635-8f28-407b-99e6-cb8a3f869117\") " pod="openshift-monitoring/metrics-server-7557bdc5fb-gmwtd"
Jul 02 13:15:55.178654 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.178536    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/886e2635-8f28-407b-99e6-cb8a3f869117-secret-metrics-server-tls\") pod \"metrics-server-7557bdc5fb-gmwtd\" (UID: \"886e2635-8f28-407b-99e6-cb8a3f869117\") " pod="openshift-monitoring/metrics-server-7557bdc5fb-gmwtd"
Jul 02 13:15:55.178654 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.178567    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/886e2635-8f28-407b-99e6-cb8a3f869117-audit-log\") pod \"metrics-server-7557bdc5fb-gmwtd\" (UID: \"886e2635-8f28-407b-99e6-cb8a3f869117\") " pod="openshift-monitoring/metrics-server-7557bdc5fb-gmwtd"
Jul 02 13:15:55.178654 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.178598    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/886e2635-8f28-407b-99e6-cb8a3f869117-metrics-server-audit-profiles\") pod \"metrics-server-7557bdc5fb-gmwtd\" (UID: \"886e2635-8f28-407b-99e6-cb8a3f869117\") " pod="openshift-monitoring/metrics-server-7557bdc5fb-gmwtd"
Jul 02 13:15:55.179548 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.179115    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/886e2635-8f28-407b-99e6-cb8a3f869117-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7557bdc5fb-gmwtd\" (UID: \"886e2635-8f28-407b-99e6-cb8a3f869117\") " pod="openshift-monitoring/metrics-server-7557bdc5fb-gmwtd"
Jul 02 13:15:55.179548 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.179505    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/886e2635-8f28-407b-99e6-cb8a3f869117-audit-log\") pod \"metrics-server-7557bdc5fb-gmwtd\" (UID: \"886e2635-8f28-407b-99e6-cb8a3f869117\") " pod="openshift-monitoring/metrics-server-7557bdc5fb-gmwtd"
Jul 02 13:15:55.179548 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.179498    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/886e2635-8f28-407b-99e6-cb8a3f869117-metrics-server-audit-profiles\") pod \"metrics-server-7557bdc5fb-gmwtd\" (UID: \"886e2635-8f28-407b-99e6-cb8a3f869117\") " pod="openshift-monitoring/metrics-server-7557bdc5fb-gmwtd"
Jul 02 13:15:55.181657 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.181533    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 13:15:55.182148 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.182116    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/886e2635-8f28-407b-99e6-cb8a3f869117-secret-metrics-server-tls\") pod \"metrics-server-7557bdc5fb-gmwtd\" (UID: \"886e2635-8f28-407b-99e6-cb8a3f869117\") " pod="openshift-monitoring/metrics-server-7557bdc5fb-gmwtd"
Jul 02 13:15:55.182241 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.182165    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/886e2635-8f28-407b-99e6-cb8a3f869117-client-ca-bundle\") pod \"metrics-server-7557bdc5fb-gmwtd\" (UID: \"886e2635-8f28-407b-99e6-cb8a3f869117\") " pod="openshift-monitoring/metrics-server-7557bdc5fb-gmwtd"
Jul 02 13:15:55.182792 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.182774    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/886e2635-8f28-407b-99e6-cb8a3f869117-secret-metrics-server-client-certs\") pod \"metrics-server-7557bdc5fb-gmwtd\" (UID: \"886e2635-8f28-407b-99e6-cb8a3f869117\") " pod="openshift-monitoring/metrics-server-7557bdc5fb-gmwtd"
Jul 02 13:15:55.187476 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.187451    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmvc4\" (UniqueName: \"kubernetes.io/projected/886e2635-8f28-407b-99e6-cb8a3f869117-kube-api-access-fmvc4\") pod \"metrics-server-7557bdc5fb-gmwtd\" (UID: \"886e2635-8f28-407b-99e6-cb8a3f869117\") " pod="openshift-monitoring/metrics-server-7557bdc5fb-gmwtd"
Jul 02 13:15:55.192697 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.192673    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 13:15:55.203154 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.203102    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2r82\" (UniqueName: \"kubernetes.io/projected/159f78cc-1fe0-40ca-a0cd-92254294389c-kube-api-access-s2r82\") pod \"network-check-target-tx24z\" (UID: \"159f78cc-1fe0-40ca-a0cd-92254294389c\") " pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:15:55.243353 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.243322    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zjxr6\""
Jul 02 13:15:55.248084 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.248054    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-drw6p\""
Jul 02 13:15:55.251449 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.251422    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:15:55.256470 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.256442    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xzl6q"
Jul 02 13:15:55.293015 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.292763    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7557bdc5fb-gmwtd"
Jul 02 13:15:55.490587 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.490254    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tkws4"
Jul 02 13:15:55.503883 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.503769    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-76d8988fc8-tpstw"]
Jul 02 13:15:55.509410 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.509118    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-76d8988fc8-tpstw"
Jul 02 13:15:55.527906 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.526886    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 02 13:15:55.527906 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.527220    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 02 13:15:55.527906 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.527403    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 02 13:15:55.527906 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.527565    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 02 13:15:55.527906 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.527761    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 02 13:15:55.537780 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.537567    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:55.540470 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.539284    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-27dzb\""
Jul 02 13:15:55.542479 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.542399    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-76d8988fc8-tpstw"]
Jul 02 13:15:55.552910 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.552680    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 02 13:15:55.586541 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.586491    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06e92769-f4b3-414a-b23c-ef41c5aa5af6-serving-certs-ca-bundle\") pod \"telemeter-client-76d8988fc8-tpstw\" (UID: \"06e92769-f4b3-414a-b23c-ef41c5aa5af6\") " pod="openshift-monitoring/telemeter-client-76d8988fc8-tpstw"
Jul 02 13:15:55.586760 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.586741    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/06e92769-f4b3-414a-b23c-ef41c5aa5af6-secret-telemeter-client\") pod \"telemeter-client-76d8988fc8-tpstw\" (UID: \"06e92769-f4b3-414a-b23c-ef41c5aa5af6\") " pod="openshift-monitoring/telemeter-client-76d8988fc8-tpstw"
Jul 02 13:15:55.586871 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.586859    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/06e92769-f4b3-414a-b23c-ef41c5aa5af6-federate-client-tls\") pod \"telemeter-client-76d8988fc8-tpstw\" (UID: \"06e92769-f4b3-414a-b23c-ef41c5aa5af6\") " pod="openshift-monitoring/telemeter-client-76d8988fc8-tpstw"
Jul 02 13:15:55.586992 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.586973    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06e92769-f4b3-414a-b23c-ef41c5aa5af6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76d8988fc8-tpstw\" (UID: \"06e92769-f4b3-414a-b23c-ef41c5aa5af6\") " pod="openshift-monitoring/telemeter-client-76d8988fc8-tpstw"
Jul 02 13:15:55.587189 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.587176    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06e92769-f4b3-414a-b23c-ef41c5aa5af6-metrics-client-ca\") pod \"telemeter-client-76d8988fc8-tpstw\" (UID: \"06e92769-f4b3-414a-b23c-ef41c5aa5af6\") " pod="openshift-monitoring/telemeter-client-76d8988fc8-tpstw"
Jul 02 13:15:55.587331 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.587315    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06e92769-f4b3-414a-b23c-ef41c5aa5af6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76d8988fc8-tpstw\" (UID: \"06e92769-f4b3-414a-b23c-ef41c5aa5af6\") " pod="openshift-monitoring/telemeter-client-76d8988fc8-tpstw"
Jul 02 13:15:55.587500 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.587473    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmqll\" (UniqueName: \"kubernetes.io/projected/06e92769-f4b3-414a-b23c-ef41c5aa5af6-kube-api-access-bmqll\") pod \"telemeter-client-76d8988fc8-tpstw\" (UID: \"06e92769-f4b3-414a-b23c-ef41c5aa5af6\") " pod="openshift-monitoring/telemeter-client-76d8988fc8-tpstw"
Jul 02 13:15:55.588049 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.588029    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/06e92769-f4b3-414a-b23c-ef41c5aa5af6-telemeter-client-tls\") pod \"telemeter-client-76d8988fc8-tpstw\" (UID: \"06e92769-f4b3-414a-b23c-ef41c5aa5af6\") " pod="openshift-monitoring/telemeter-client-76d8988fc8-tpstw"
Jul 02 13:15:55.597186 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.595463    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pjp9n" event={"ID":"a5e06a98-36dc-49b0-959c-d646b9802308","Type":"ContainerStarted","Data":"b63a3e6f3bec4ee373fd9903b3ce2f9e1719b73310637796f12744543e260e85"}
Jul 02 13:15:55.597186 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.595524    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pjp9n" event={"ID":"a5e06a98-36dc-49b0-959c-d646b9802308","Type":"ContainerStarted","Data":"046addb00661d2924046058570fe88742ce8c35092ede1ca6fd23715772f937c"}
Jul 02 13:15:55.640151 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.639987    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pjp9n" podStartSLOduration=2.917464871 podStartE2EDuration="5.639963038s" totalImagesPullingTime="2.638010827s" totalInitContainerRuntime="84.48734ms" isStatefulPod=true podCreationTimestamp="2026-07-02 13:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:55.620305294 +0000 UTC m=+66.848216505" watchObservedRunningTime="2026-07-02 13:15:55.639963038 +0000 UTC m=+66.867874213"
Jul 02 13:15:55.641328 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.641298    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7557bdc5fb-gmwtd"]
Jul 02 13:15:55.646511 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:15:55.646470    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod886e2635_8f28_407b_99e6_cb8a3f869117.slice/crio-adb53cd560df2b891a4371c79ece93e342eda2f0fc730d211c5363d65ba20e28 WatchSource:0}: Error finding container adb53cd560df2b891a4371c79ece93e342eda2f0fc730d211c5363d65ba20e28: Status 404 returned error can't find the container with id adb53cd560df2b891a4371c79ece93e342eda2f0fc730d211c5363d65ba20e28
Jul 02 13:15:55.652193 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.652160    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9fpt5"]
Jul 02 13:15:55.659534 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:15:55.659105    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa11a9f_ca92_4214_8e84_0b9c48a8337c.slice/crio-3b1e154e80aa02bdb50968bda95953b78926fcb3c436c4dd50d64d8538cd09f2 WatchSource:0}: Error finding container 3b1e154e80aa02bdb50968bda95953b78926fcb3c436c4dd50d64d8538cd09f2: Status 404 returned error can't find the container with id 3b1e154e80aa02bdb50968bda95953b78926fcb3c436c4dd50d64d8538cd09f2
Jul 02 13:15:55.688858 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.688817    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/06e92769-f4b3-414a-b23c-ef41c5aa5af6-telemeter-client-tls\") pod \"telemeter-client-76d8988fc8-tpstw\" (UID: \"06e92769-f4b3-414a-b23c-ef41c5aa5af6\") " pod="openshift-monitoring/telemeter-client-76d8988fc8-tpstw"
Jul 02 13:15:55.689019 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.688999    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06e92769-f4b3-414a-b23c-ef41c5aa5af6-serving-certs-ca-bundle\") pod \"telemeter-client-76d8988fc8-tpstw\" (UID: \"06e92769-f4b3-414a-b23c-ef41c5aa5af6\") " pod="openshift-monitoring/telemeter-client-76d8988fc8-tpstw"
Jul 02 13:15:55.689101 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.689037    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/06e92769-f4b3-414a-b23c-ef41c5aa5af6-secret-telemeter-client\") pod \"telemeter-client-76d8988fc8-tpstw\" (UID: \"06e92769-f4b3-414a-b23c-ef41c5aa5af6\") " pod="openshift-monitoring/telemeter-client-76d8988fc8-tpstw"
Jul 02 13:15:55.689101 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.689062    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/06e92769-f4b3-414a-b23c-ef41c5aa5af6-federate-client-tls\") pod \"telemeter-client-76d8988fc8-tpstw\" (UID: \"06e92769-f4b3-414a-b23c-ef41c5aa5af6\") " pod="openshift-monitoring/telemeter-client-76d8988fc8-tpstw"
Jul 02 13:15:55.689101 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.689089    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06e92769-f4b3-414a-b23c-ef41c5aa5af6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76d8988fc8-tpstw\" (UID: \"06e92769-f4b3-414a-b23c-ef41c5aa5af6\") " pod="openshift-monitoring/telemeter-client-76d8988fc8-tpstw"
Jul 02 13:15:55.689236 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.689179    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06e92769-f4b3-414a-b23c-ef41c5aa5af6-metrics-client-ca\") pod \"telemeter-client-76d8988fc8-tpstw\" (UID: \"06e92769-f4b3-414a-b23c-ef41c5aa5af6\") " pod="openshift-monitoring/telemeter-client-76d8988fc8-tpstw"
Jul 02 13:15:55.689236 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.689207    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06e92769-f4b3-414a-b23c-ef41c5aa5af6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76d8988fc8-tpstw\" (UID: \"06e92769-f4b3-414a-b23c-ef41c5aa5af6\") " pod="openshift-monitoring/telemeter-client-76d8988fc8-tpstw"
Jul 02 13:15:55.689327 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.689247    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmqll\" (UniqueName: \"kubernetes.io/projected/06e92769-f4b3-414a-b23c-ef41c5aa5af6-kube-api-access-bmqll\") pod \"telemeter-client-76d8988fc8-tpstw\" (UID: \"06e92769-f4b3-414a-b23c-ef41c5aa5af6\") " pod="openshift-monitoring/telemeter-client-76d8988fc8-tpstw"
Jul 02 13:15:55.690813 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.690786    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06e92769-f4b3-414a-b23c-ef41c5aa5af6-serving-certs-ca-bundle\") pod \"telemeter-client-76d8988fc8-tpstw\" (UID: \"06e92769-f4b3-414a-b23c-ef41c5aa5af6\") " pod="openshift-monitoring/telemeter-client-76d8988fc8-tpstw"
Jul 02 13:15:55.692219 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.692179    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06e92769-f4b3-414a-b23c-ef41c5aa5af6-metrics-client-ca\") pod \"telemeter-client-76d8988fc8-tpstw\" (UID: \"06e92769-f4b3-414a-b23c-ef41c5aa5af6\") " pod="openshift-monitoring/telemeter-client-76d8988fc8-tpstw"
Jul 02 13:15:55.694289 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.694245    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06e92769-f4b3-414a-b23c-ef41c5aa5af6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76d8988fc8-tpstw\" (UID: \"06e92769-f4b3-414a-b23c-ef41c5aa5af6\") " pod="openshift-monitoring/telemeter-client-76d8988fc8-tpstw"
Jul 02 13:15:55.695755 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.695725    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-cd8656b8-fq7jd"]
Jul 02 13:15:55.695901 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.695876    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/06e92769-f4b3-414a-b23c-ef41c5aa5af6-secret-telemeter-client\") pod \"telemeter-client-76d8988fc8-tpstw\" (UID: \"06e92769-f4b3-414a-b23c-ef41c5aa5af6\") " pod="openshift-monitoring/telemeter-client-76d8988fc8-tpstw"
Jul 02 13:15:55.696107 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.696080    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06e92769-f4b3-414a-b23c-ef41c5aa5af6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76d8988fc8-tpstw\" (UID: \"06e92769-f4b3-414a-b23c-ef41c5aa5af6\") " pod="openshift-monitoring/telemeter-client-76d8988fc8-tpstw"
Jul 02 13:15:55.700141 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.696450    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/06e92769-f4b3-414a-b23c-ef41c5aa5af6-telemeter-client-tls\") pod \"telemeter-client-76d8988fc8-tpstw\" (UID: \"06e92769-f4b3-414a-b23c-ef41c5aa5af6\") " pod="openshift-monitoring/telemeter-client-76d8988fc8-tpstw"
Jul 02 13:15:55.701354 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.701306    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/06e92769-f4b3-414a-b23c-ef41c5aa5af6-federate-client-tls\") pod \"telemeter-client-76d8988fc8-tpstw\" (UID: \"06e92769-f4b3-414a-b23c-ef41c5aa5af6\") " pod="openshift-monitoring/telemeter-client-76d8988fc8-tpstw"
Jul 02 13:15:55.702954 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.702907    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmqll\" (UniqueName: \"kubernetes.io/projected/06e92769-f4b3-414a-b23c-ef41c5aa5af6-kube-api-access-bmqll\") pod \"telemeter-client-76d8988fc8-tpstw\" (UID: \"06e92769-f4b3-414a-b23c-ef41c5aa5af6\") " pod="openshift-monitoring/telemeter-client-76d8988fc8-tpstw"
Jul 02 13:15:55.854744 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.854710    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-76d8988fc8-tpstw"
Jul 02 13:15:55.936639 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.936553    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xzl6q"]
Jul 02 13:15:55.940860 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:55.940835    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tx24z"]
Jul 02 13:15:56.354735 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.354629    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:56.360853 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.360576    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 13:15:56.364275 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.364245    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.367436 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.367408    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 02 13:15:56.367645 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.367624    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 02 13:15:56.367714 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.367678    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 02 13:15:56.367914 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.367893    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 02 13:15:56.367914 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.367911    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jh8sh\""
Jul 02 13:15:56.369100 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.368110    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 02 13:15:56.369100 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.368157    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 02 13:15:56.369100 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.368454    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 02 13:15:56.369100 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.368475    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 02 13:15:56.369100 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.368700    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 02 13:15:56.369100 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.368815    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 02 13:15:56.369100 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.368960    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9m5jmmh8jgtj5\""
Jul 02 13:15:56.369100 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.369080    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 02 13:15:56.369579 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.369268    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 02 13:15:56.369579 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.369476    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 02 13:15:56.372953 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.372843    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 02 13:15:56.383247 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.383216    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 13:15:56.397875 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.397747    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.397875 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.397791    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.397875 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.397819    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5772ab56-d521-4af5-8c02-4d017569eb98-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.398139 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.397903    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5772ab56-d521-4af5-8c02-4d017569eb98-config-out\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.398139 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.397971    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-config\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.398139 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.398003    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.398139 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.398039    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.398139 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.398065    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.398139 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.398090    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.398139 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.398126    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.398504 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.398166    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.398504 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.398192    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.398504 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.398216    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.398504 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.398243    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-web-config\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.398504 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.398296    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.398504 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.398319    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhxkg\" (UniqueName: \"kubernetes.io/projected/5772ab56-d521-4af5-8c02-4d017569eb98-kube-api-access-vhxkg\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.398504 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.398350    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.398504 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.398415    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5772ab56-d521-4af5-8c02-4d017569eb98-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.398504 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.398444    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.398504 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.398471    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.485428 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:15:56.485360    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod159f78cc_1fe0_40ca_a0cd_92254294389c.slice/crio-54985fe16beca8090aeda8c480ce77f794b9cfaaec140b06995aa33dcefb9676 WatchSource:0}: Error finding container 54985fe16beca8090aeda8c480ce77f794b9cfaaec140b06995aa33dcefb9676: Status 404 returned error can't find the container with id 54985fe16beca8090aeda8c480ce77f794b9cfaaec140b06995aa33dcefb9676
Jul 02 13:15:56.487331 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:15:56.487078    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cae18b4_8f5b_4ff7_a73f_fc668af0a663.slice/crio-c136bb4961e4100a652dbb9fab64a0cc828ce559719d720de308d0bf512716da WatchSource:0}: Error finding container c136bb4961e4100a652dbb9fab64a0cc828ce559719d720de308d0bf512716da: Status 404 returned error can't find the container with id c136bb4961e4100a652dbb9fab64a0cc828ce559719d720de308d0bf512716da
Jul 02 13:15:56.499772 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.499557    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.499772 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.499601    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhxkg\" (UniqueName: \"kubernetes.io/projected/5772ab56-d521-4af5-8c02-4d017569eb98-kube-api-access-vhxkg\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.499772 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.499628    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.499772 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.499660    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5772ab56-d521-4af5-8c02-4d017569eb98-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.499772 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.499690    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.499772 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.499715    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.500484 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.500406    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.500484 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.500448    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.500636 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.500609    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.501188 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.501136    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.501342 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.501311    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5772ab56-d521-4af5-8c02-4d017569eb98-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.501457 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.501371    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5772ab56-d521-4af5-8c02-4d017569eb98-config-out\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.501457 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.501429    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-config\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.501457 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.501453    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.501620 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.501498    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.501620 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.501523    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.501620 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.501548    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.501620 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.501585    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.501816 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.501629    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.501816 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.501655    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.501816 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.501677    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.501816 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.501702    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-web-config\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.502726 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.502464    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.507162 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.506677    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.507162 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.507071    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.511547 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.507914    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.511547 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.508699    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.511547 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.509073    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-config\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.511547 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.509346    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.511547 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.509560    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5772ab56-d521-4af5-8c02-4d017569eb98-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.511547 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.510045    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.511547 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.510110    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.511547 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.510746    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-web-config\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.511547 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.510977    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.512099 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.511583    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.512743 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.512679    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.514610 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.514558    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.515418 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.515276    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5772ab56-d521-4af5-8c02-4d017569eb98-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.515797 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.515688    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhxkg\" (UniqueName: \"kubernetes.io/projected/5772ab56-d521-4af5-8c02-4d017569eb98-kube-api-access-vhxkg\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.517013 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.516965    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5772ab56-d521-4af5-8c02-4d017569eb98-config-out\") pod \"prometheus-k8s-0\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.605613 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.605446    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tx24z" event={"ID":"159f78cc-1fe0-40ca-a0cd-92254294389c","Type":"ContainerStarted","Data":"54985fe16beca8090aeda8c480ce77f794b9cfaaec140b06995aa33dcefb9676"}
Jul 02 13:15:56.607794 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.607670    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-cd8656b8-fq7jd" event={"ID":"6e4c183e-20bc-408d-b975-bc6a9aafe76c","Type":"ContainerStarted","Data":"13b996913fa89a7e6c0fcf4c7caec4e98e1cd33536750fb5a266a777ddfd66ce"}
Jul 02 13:15:56.611148 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.611108    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c2cc0d64-e3d1-429e-8eda-f07d72ceab95","Type":"ContainerStarted","Data":"ada7017e79d620eb721290bfbb767d8f1ad3f37ffb4ca5e041322e33e2401d50"}
Jul 02 13:15:56.611261 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.611195    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c2cc0d64-e3d1-429e-8eda-f07d72ceab95","Type":"ContainerStarted","Data":"b355077d48fb0a743a04f5bd919565534855f61be8801b965a697754b2cb9a20"}
Jul 02 13:15:56.611261 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.611240    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c2cc0d64-e3d1-429e-8eda-f07d72ceab95","Type":"ContainerStarted","Data":"b79d203129c1d4eacd6b84b1c22adb9619997a37836a1e01551d22d25007c1e9"}
Jul 02 13:15:56.618144 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.618060    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9fpt5" event={"ID":"bfa11a9f-ca92-4214-8e84-0b9c48a8337c","Type":"ContainerStarted","Data":"3b1e154e80aa02bdb50968bda95953b78926fcb3c436c4dd50d64d8538cd09f2"}
Jul 02 13:15:56.619623 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.619559    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7557bdc5fb-gmwtd" event={"ID":"886e2635-8f28-407b-99e6-cb8a3f869117","Type":"ContainerStarted","Data":"adb53cd560df2b891a4371c79ece93e342eda2f0fc730d211c5363d65ba20e28"}
Jul 02 13:15:56.622118 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.622059    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xzl6q" event={"ID":"0cae18b4-8f5b-4ff7-a73f-fc668af0a663","Type":"ContainerStarted","Data":"c136bb4961e4100a652dbb9fab64a0cc828ce559719d720de308d0bf512716da"}
Jul 02 13:15:56.681246 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.681021    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:15:56.687469 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.686867    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-76d8988fc8-tpstw"]
Jul 02 13:15:56.693318 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:15:56.693288    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e92769_f4b3_414a_b23c_ef41c5aa5af6.slice/crio-ec1075eb6812e71659f4b79e288d4c027f3d8d77cb94be28a942fc84f3f65d03 WatchSource:0}: Error finding container ec1075eb6812e71659f4b79e288d4c027f3d8d77cb94be28a942fc84f3f65d03: Status 404 returned error can't find the container with id ec1075eb6812e71659f4b79e288d4c027f3d8d77cb94be28a942fc84f3f65d03
Jul 02 13:15:56.925782 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:56.925749    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 13:15:56.926276 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:15:56.926247    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5772ab56_d521_4af5_8c02_4d017569eb98.slice/crio-3c595b7396770c74d5966520f7ca141c075a8410d71604e9ef2bc91ccae3b539 WatchSource:0}: Error finding container 3c595b7396770c74d5966520f7ca141c075a8410d71604e9ef2bc91ccae3b539: Status 404 returned error can't find the container with id 3c595b7396770c74d5966520f7ca141c075a8410d71604e9ef2bc91ccae3b539
Jul 02 13:15:57.163057 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:57.163022    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:57.338492 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:57.338305    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:57.633925 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:57.633730    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x" event={"ID":"5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3","Type":"ContainerStarted","Data":"d18830e8c505ab1868b67dd6611353d7cda60a3d0eed1c2114da525c2ee9f5fc"}
Jul 02 13:15:57.633925 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:57.633816    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x" event={"ID":"5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3","Type":"ContainerStarted","Data":"34e245dadd52f27963647cf4f0eae8f676ab597ff343a6cf8ad3469bc211b787"}
Jul 02 13:15:57.633925 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:57.633861    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x" event={"ID":"5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3","Type":"ContainerStarted","Data":"39d47e2deacee0eed5feb138e8421a88bacf817a2d13c2220794919eee513a06"}
Jul 02 13:15:57.644221 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:57.644071    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c2cc0d64-e3d1-429e-8eda-f07d72ceab95","Type":"ContainerStarted","Data":"893620c671ff83926e981be6776e2888269b028d1f37836eb5dc4a7da1cde2bd"}
Jul 02 13:15:57.644221 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:57.644155    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c2cc0d64-e3d1-429e-8eda-f07d72ceab95","Type":"ContainerStarted","Data":"09c787349750c7e5a9bd9a795dbee3261e5c3ae3b7dde349cd937e622da77084"}
Jul 02 13:15:57.650301 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:57.649066    2583 generic.go:356] "Generic (PLEG): container finished" podID="5772ab56-d521-4af5-8c02-4d017569eb98" containerID="e344ae5150a86bb2342e9cd4ef14f6c9902cfbd4fe85012ce50b9b4bdb7ae7f0" exitCode=0
Jul 02 13:15:57.650301 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:57.649143    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5772ab56-d521-4af5-8c02-4d017569eb98","Type":"ContainerDied","Data":"e344ae5150a86bb2342e9cd4ef14f6c9902cfbd4fe85012ce50b9b4bdb7ae7f0"}
Jul 02 13:15:57.650301 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:57.649214    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5772ab56-d521-4af5-8c02-4d017569eb98","Type":"ContainerStarted","Data":"3c595b7396770c74d5966520f7ca141c075a8410d71604e9ef2bc91ccae3b539"}
Jul 02 13:15:57.654188 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:57.654112    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76d8988fc8-tpstw" event={"ID":"06e92769-f4b3-414a-b23c-ef41c5aa5af6","Type":"ContainerStarted","Data":"ec1075eb6812e71659f4b79e288d4c027f3d8d77cb94be28a942fc84f3f65d03"}
Jul 02 13:15:58.145062 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:58.144969    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:58.246453 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:58.246335    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:58.319790 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:58.319676    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:58.392490 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:58.392452    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:58.494102 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:58.494006    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:58.592593 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:58.592559    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:59.807191 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:59.807140    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:15:59.908154 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:15:59.908085    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:01.117893 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:01.117864    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:01.220759 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:01.220694    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:03.211296 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:03.211175    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:03.335570 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:03.335532    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:05.708306 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:05.706204    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c2cc0d64-e3d1-429e-8eda-f07d72ceab95","Type":"ContainerStarted","Data":"097db36fa3ba4890896b738e466d6f28891071b2ef3b656c0b7f069f08751d47"}
Jul 02 13:16:05.719456 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:05.718842    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9fpt5" event={"ID":"bfa11a9f-ca92-4214-8e84-0b9c48a8337c","Type":"ContainerStarted","Data":"db555f7e7b953b825b57f1d52bb8a8368efbb13390c69795491e47b8cebfa341"}
Jul 02 13:16:05.727762 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:05.727137    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7557bdc5fb-gmwtd" event={"ID":"886e2635-8f28-407b-99e6-cb8a3f869117","Type":"ContainerStarted","Data":"ac1a67705fbc4c60c355efbe29ae1df85d71abb1d14ba756127f1676dc8675f7"}
Jul 02 13:16:05.736489 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:05.736308    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5772ab56-d521-4af5-8c02-4d017569eb98","Type":"ContainerStarted","Data":"580d6f6bb79535caffffba94bf5b4ddeca1c77f84377f47815d14cd12f362eee"}
Jul 02 13:16:05.736489 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:05.736422    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5772ab56-d521-4af5-8c02-4d017569eb98","Type":"ContainerStarted","Data":"b411e9a05b50c7209cbf1d2c1f8f438ec6e4457401b00ae96a90f2687c2d91d7"}
Jul 02 13:16:05.749041 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:05.748869    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76d8988fc8-tpstw" event={"ID":"06e92769-f4b3-414a-b23c-ef41c5aa5af6","Type":"ContainerStarted","Data":"81cc62109829ac403c8ef738c62cd347ce1299a2947a7d9fcc861afca2743a46"}
Jul 02 13:16:05.749041 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:05.748942    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76d8988fc8-tpstw" event={"ID":"06e92769-f4b3-414a-b23c-ef41c5aa5af6","Type":"ContainerStarted","Data":"85f98b90046e0b8816761ee40baa82dff64db08e02fb1a810688825d1c1b0070"}
Jul 02 13:16:05.764060 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:05.762675    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x" event={"ID":"5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3","Type":"ContainerStarted","Data":"5e83c40f093dbd470226d594ba01dd3b0cb9199df1f761b0183e094ed911ee63"}
Jul 02 13:16:05.764060 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:05.762769    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x" event={"ID":"5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3","Type":"ContainerStarted","Data":"d54e007f8a8e0880a7cf5b3cd2cc44355a748934d28abb0fcaa0ce0fbc86b24d"}
Jul 02 13:16:05.764321 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:05.764193    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x"
Jul 02 13:16:05.779258 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:05.779149    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xzl6q" event={"ID":"0cae18b4-8f5b-4ff7-a73f-fc668af0a663","Type":"ContainerStarted","Data":"dfb80214e1e084ce70cf328db739a64dae08d949fd80fa340da2d4937b1fa868"}
Jul 02 13:16:05.779258 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:05.779208    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xzl6q" event={"ID":"0cae18b4-8f5b-4ff7-a73f-fc668af0a663","Type":"ContainerStarted","Data":"f273f7c9f03db03cc6c3706cf08175033da6aca8aa7a9ebb088902e2700c1835"}
Jul 02 13:16:05.782170 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:05.781576    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tx24z" event={"ID":"159f78cc-1fe0-40ca-a0cd-92254294389c","Type":"ContainerStarted","Data":"576c56357fd8ebd6e71971770c27e559dde449666372ec7af041d67c090db830"}
Jul 02 13:16:05.782170 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:05.782106    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:16:05.786296 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:05.784251    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-cd8656b8-fq7jd" event={"ID":"6e4c183e-20bc-408d-b975-bc6a9aafe76c","Type":"ContainerStarted","Data":"b2dd5250703ad6ff0d9bb62b09788ffc20cd4079357e84fa73fda86801e0040d"}
Jul 02 13:16:05.786296 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:05.785158    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-cd8656b8-fq7jd"
Jul 02 13:16:05.791941 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:05.791292    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.527514017 podStartE2EDuration="14.791272064s" totalImagesPullingTime="11.244472534s" totalInitContainerRuntime="19.285513ms" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:16:05.758819418 +0000 UTC m=+76.986730594" watchObservedRunningTime="2026-07-02 13:16:05.791272064 +0000 UTC m=+77.019183236"
Jul 02 13:16:05.796429 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:05.796347    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-cd8656b8-fq7jd"
Jul 02 13:16:05.802251 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:05.802155    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x"
Jul 02 13:16:05.834157 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:05.829906    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9fpt5" podStartSLOduration=67.299129559 podStartE2EDuration="1m16.82988328s" totalImagesPullingTime="9.530753721s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:16:05.791780704 +0000 UTC m=+77.019691878" watchObservedRunningTime="2026-07-02 13:16:05.82988328 +0000 UTC m=+77.057794457"
Jul 02 13:16:05.863411 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:05.863066    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-76d8988fc8-tpstw" podStartSLOduration=2.372150365 podStartE2EDuration="10.863042987s" totalImagesPullingTime="8.490892622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:16:05.833984436 +0000 UTC m=+77.061895611" watchObservedRunningTime="2026-07-02 13:16:05.863042987 +0000 UTC m=+77.090954163"
Jul 02 13:16:05.894130 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:05.893661    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-cd8656b8-fq7jd" podStartSLOduration=2.414924095 podStartE2EDuration="11.893638415s" totalImagesPullingTime="9.47871432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:16:05.89154846 +0000 UTC m=+77.119459670" watchObservedRunningTime="2026-07-02 13:16:05.893638415 +0000 UTC m=+77.121549594"
Jul 02 13:16:05.894130 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:05.893767    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7557bdc5fb-gmwtd" podStartSLOduration=2.3586184709999998 podStartE2EDuration="11.893758191s" totalImagesPullingTime="9.53513972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:16:05.863823195 +0000 UTC m=+77.091734370" watchObservedRunningTime="2026-07-02 13:16:05.893758191 +0000 UTC m=+77.121669368"
Jul 02 13:16:05.921206 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:05.918936    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xzl6q" podStartSLOduration=69.491162176 podStartE2EDuration="1m16.918912916s" totalImagesPullingTime="7.42775074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:16:05.916766415 +0000 UTC m=+77.144677590" watchObservedRunningTime="2026-07-02 13:16:05.918912916 +0000 UTC m=+77.146824093"
Jul 02 13:16:05.944073 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:05.943314    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tx24z" podStartSLOduration=68.234878741 podStartE2EDuration="1m16.943291973s" totalImagesPullingTime="8.708413232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:16:05.942064069 +0000 UTC m=+77.169975242" watchObservedRunningTime="2026-07-02 13:16:05.943291973 +0000 UTC m=+77.171203150"
Jul 02 13:16:05.968433 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:05.968253    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x" podStartSLOduration=1.8994495329999999 podStartE2EDuration="13.968234456s" totalImagesPullingTime="12.068784923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:16:05.966362903 +0000 UTC m=+77.194274078" watchObservedRunningTime="2026-07-02 13:16:05.968234456 +0000 UTC m=+77.196145629"
Jul 02 13:16:06.040122 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:06.040072    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:06.570913 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:06.570878    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:06.688204 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:06.688160    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:06.797081 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:06.797044    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5772ab56-d521-4af5-8c02-4d017569eb98","Type":"ContainerStarted","Data":"69dbdde8164c3e3c5eb3946a35e9f6dca70b2785162bb3498bcb7f4fb78a7082"}
Jul 02 13:16:06.797685 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:06.797653    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5772ab56-d521-4af5-8c02-4d017569eb98","Type":"ContainerStarted","Data":"76c8f8c577ab675b334167508900629fa74ac333a5e4e3a003d80938b77bc813"}
Jul 02 13:16:06.797823 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:06.797716    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5772ab56-d521-4af5-8c02-4d017569eb98","Type":"ContainerStarted","Data":"c7d3dadeb32b528bae0c69d223f0ed263240d57a73e6df86a17e331733bddf60"}
Jul 02 13:16:06.797823 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:06.797758    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5772ab56-d521-4af5-8c02-4d017569eb98","Type":"ContainerStarted","Data":"1c586f4447e899f63eb93699c7377ca8579b1ab5e2f9466f4babbfb0bb1275b4"}
Jul 02 13:16:06.801824 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:06.801795    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76d8988fc8-tpstw" event={"ID":"06e92769-f4b3-414a-b23c-ef41c5aa5af6","Type":"ContainerStarted","Data":"c74efe9e22f848d59524b45503ff7a758fba296122df1165f8afe5be0e17ed45"}
Jul 02 13:16:06.805733 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:06.805692    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57f4dddd4d-p6t8x" event={"ID":"5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3","Type":"ContainerStarted","Data":"9d59c3ea20a3fed0ad8efc9d66e77dd1ce8b455b92c0cdec154f26024e71376f"}
Jul 02 13:16:06.836023 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:06.835608    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.21061451 podStartE2EDuration="10.835584099s" totalImagesPullingTime="7.597307457s" totalInitContainerRuntime="27.662132ms" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:16:06.831709059 +0000 UTC m=+78.059620234" watchObservedRunningTime="2026-07-02 13:16:06.835584099 +0000 UTC m=+78.063495274"
Jul 02 13:16:07.285124 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:07.285088    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:07.731969 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:07.731935    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:07.804058 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:07.804016    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:08.274007 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:08.273950    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dd8n6_bc66f32c-20f9-44d2-b58e-5e3b8ede1958/serve-healthcheck-canary/0.log"
Jul 02 13:16:08.917292 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:08.917256    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:08.992325 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:08.992290    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:11.681935 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:11.681783    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:16:13.293799 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:13.293760    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:13.576467 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:13.576368    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:14.336600 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:14.336565    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:14.852049 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:14.852015    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5499b6fb8f-mxb5n" event={"ID":"79b6d559-f1fe-45a4-9402-5c4008c2108b","Type":"ContainerStarted","Data":"fc87017071c92c64b31c55f834cbc7e0bcecc83516076770f862eb86f7bf8f6c"}
Jul 02 13:16:14.852224 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:14.852179    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5499b6fb8f-mxb5n"
Jul 02 13:16:14.853744 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:14.853718    2583 patch_prober.go:28] interesting pod/downloads-5499b6fb8f-mxb5n container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.12:8080/\": dial tcp 10.134.0.12:8080: connect: connection refused" start-of-body=
Jul 02 13:16:14.853864 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:14.853767    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-5499b6fb8f-mxb5n" podUID="79b6d559-f1fe-45a4-9402-5c4008c2108b" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.12:8080/\": dial tcp 10.134.0.12:8080: connect: connection refused"
Jul 02 13:16:15.293605 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:15.293562    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7557bdc5fb-gmwtd"
Jul 02 13:16:15.293807 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:15.293644    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7557bdc5fb-gmwtd"
Jul 02 13:16:15.875953 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:15.875912    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5499b6fb8f-mxb5n"
Jul 02 13:16:15.897982 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:15.897039    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5499b6fb8f-mxb5n" podStartSLOduration=2.64622514 podStartE2EDuration="25.897017532s" totalImagesPullingTime="23.250792392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:16:14.875912829 +0000 UTC m=+86.103824004" watchObservedRunningTime="2026-07-02 13:16:15.897017532 +0000 UTC m=+87.124928706"
Jul 02 13:16:19.563055 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:19.563013    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:19.666334 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:19.666296    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:19.768337 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:19.768299    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:21.496689 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:21.496650    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:21.600633 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:21.600587    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:22.690372 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:22.690342    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:22.791416 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:22.791371    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:22.867484 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:22.867434    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:27.905548 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:27.905513    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:28.353193 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:28.353157    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:35.298674 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:35.298645    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7557bdc5fb-gmwtd"
Jul 02 13:16:35.302561 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:35.302538    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7557bdc5fb-gmwtd"
Jul 02 13:16:37.813532 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:37.813499    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tx24z"
Jul 02 13:16:38.021793 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:38.021761    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:38.122768 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:38.122735    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:49.557812 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:49.557776    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:49.659287 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:49.659253    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:52.710272 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:52.710237    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:52.771570 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:52.771537    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:56.681994 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:56.681956    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:16:56.739229 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:56.739203    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:16:57.054424 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:57.054335    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:16:57.302636 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:57.302602    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:16:57.376245 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:16:57.376217    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:06.264745 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:06.264707    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:06.342555 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:06.342520    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:10.536605 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:10.536569    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 13:17:10.537083 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:10.536946    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" containerName="alertmanager" containerID="cri-o://b79d203129c1d4eacd6b84b1c22adb9619997a37836a1e01551d22d25007c1e9" gracePeriod=120
Jul 02 13:17:10.537083 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:10.537016    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" containerName="kube-rbac-proxy-web" containerID="cri-o://ada7017e79d620eb721290bfbb767d8f1ad3f37ffb4ca5e041322e33e2401d50" gracePeriod=120
Jul 02 13:17:10.537083 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:10.537020    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" containerName="kube-rbac-proxy-metric" containerID="cri-o://893620c671ff83926e981be6776e2888269b028d1f37836eb5dc4a7da1cde2bd" gracePeriod=120
Jul 02 13:17:10.537230 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:10.537069    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" containerName="prom-label-proxy" containerID="cri-o://097db36fa3ba4890896b738e466d6f28891071b2ef3b656c0b7f069f08751d47" gracePeriod=120
Jul 02 13:17:10.537230 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:10.537020    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" containerName="kube-rbac-proxy" containerID="cri-o://09c787349750c7e5a9bd9a795dbee3261e5c3ae3b7dde349cd937e622da77084" gracePeriod=120
Jul 02 13:17:10.537230 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:10.537170    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" containerName="config-reloader" containerID="cri-o://b355077d48fb0a743a04f5bd919565534855f61be8801b965a697754b2cb9a20" gracePeriod=120
Jul 02 13:17:11.096799 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.096769    2583 generic.go:356] "Generic (PLEG): container finished" podID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" containerID="097db36fa3ba4890896b738e466d6f28891071b2ef3b656c0b7f069f08751d47" exitCode=0
Jul 02 13:17:11.096799 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.096790    2583 generic.go:356] "Generic (PLEG): container finished" podID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" containerID="09c787349750c7e5a9bd9a795dbee3261e5c3ae3b7dde349cd937e622da77084" exitCode=0
Jul 02 13:17:11.096799 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.096798    2583 generic.go:356] "Generic (PLEG): container finished" podID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" containerID="b355077d48fb0a743a04f5bd919565534855f61be8801b965a697754b2cb9a20" exitCode=0
Jul 02 13:17:11.096799 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.096806    2583 generic.go:356] "Generic (PLEG): container finished" podID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" containerID="b79d203129c1d4eacd6b84b1c22adb9619997a37836a1e01551d22d25007c1e9" exitCode=0
Jul 02 13:17:11.097054 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.096836    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c2cc0d64-e3d1-429e-8eda-f07d72ceab95","Type":"ContainerDied","Data":"097db36fa3ba4890896b738e466d6f28891071b2ef3b656c0b7f069f08751d47"}
Jul 02 13:17:11.097054 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.096899    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c2cc0d64-e3d1-429e-8eda-f07d72ceab95","Type":"ContainerDied","Data":"09c787349750c7e5a9bd9a795dbee3261e5c3ae3b7dde349cd937e622da77084"}
Jul 02 13:17:11.097054 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.096926    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c2cc0d64-e3d1-429e-8eda-f07d72ceab95","Type":"ContainerDied","Data":"b355077d48fb0a743a04f5bd919565534855f61be8801b965a697754b2cb9a20"}
Jul 02 13:17:11.097054 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.096949    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c2cc0d64-e3d1-429e-8eda-f07d72ceab95","Type":"ContainerDied","Data":"b79d203129c1d4eacd6b84b1c22adb9619997a37836a1e01551d22d25007c1e9"}
Jul 02 13:17:11.779344 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.779320    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:11.882647 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.882611    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-secret-alertmanager-main-tls\") pod \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") "
Jul 02 13:17:11.882831 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.882652    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-cluster-tls-config\") pod \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") "
Jul 02 13:17:11.882831 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.882725    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-alertmanager-main-db\") pod \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") "
Jul 02 13:17:11.882831 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.882747    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-alertmanager-trusted-ca-bundle\") pod \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") "
Jul 02 13:17:11.882831 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.882775    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qx9q8\" (UniqueName: \"kubernetes.io/projected/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-kube-api-access-qx9q8\") pod \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") "
Jul 02 13:17:11.882831 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.882801    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-config-out\") pod \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") "
Jul 02 13:17:11.883094 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.882939    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-config-volume\") pod \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") "
Jul 02 13:17:11.883094 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.882996    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-secret-alertmanager-kube-rbac-proxy\") pod \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") "
Jul 02 13:17:11.883094 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.883039    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-metrics-client-ca\") pod \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") "
Jul 02 13:17:11.883094 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.883049    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "c2cc0d64-e3d1-429e-8eda-f07d72ceab95" (UID: "c2cc0d64-e3d1-429e-8eda-f07d72ceab95"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:17:11.883094 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.883079    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") "
Jul 02 13:17:11.883329 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.883105    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-secret-alertmanager-kube-rbac-proxy-web\") pod \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") "
Jul 02 13:17:11.883329 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.883124    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-tls-assets\") pod \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") "
Jul 02 13:17:11.883329 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.883148    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-web-config\") pod \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\" (UID: \"c2cc0d64-e3d1-429e-8eda-f07d72ceab95\") "
Jul 02 13:17:11.883515 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.883348    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "c2cc0d64-e3d1-429e-8eda-f07d72ceab95" (UID: "c2cc0d64-e3d1-429e-8eda-f07d72ceab95"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:17:11.883515 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.883457    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "c2cc0d64-e3d1-429e-8eda-f07d72ceab95" (UID: "c2cc0d64-e3d1-429e-8eda-f07d72ceab95"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:17:11.883623 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.883517    2583 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-alertmanager-main-db\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:17:11.883623 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.883539    2583 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:17:11.886268 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.886217    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "c2cc0d64-e3d1-429e-8eda-f07d72ceab95" (UID: "c2cc0d64-e3d1-429e-8eda-f07d72ceab95"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:11.886643 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.886604    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-config-out" (OuterVolumeSpecName: "config-out") pod "c2cc0d64-e3d1-429e-8eda-f07d72ceab95" (UID: "c2cc0d64-e3d1-429e-8eda-f07d72ceab95"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:17:11.886891 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.886837    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-config-volume" (OuterVolumeSpecName: "config-volume") pod "c2cc0d64-e3d1-429e-8eda-f07d72ceab95" (UID: "c2cc0d64-e3d1-429e-8eda-f07d72ceab95"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:11.887013 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.886975    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "c2cc0d64-e3d1-429e-8eda-f07d72ceab95" (UID: "c2cc0d64-e3d1-429e-8eda-f07d72ceab95"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:11.887118 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.887007    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-kube-api-access-qx9q8" (OuterVolumeSpecName: "kube-api-access-qx9q8") pod "c2cc0d64-e3d1-429e-8eda-f07d72ceab95" (UID: "c2cc0d64-e3d1-429e-8eda-f07d72ceab95"). InnerVolumeSpecName "kube-api-access-qx9q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:17:11.887118 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.887088    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c2cc0d64-e3d1-429e-8eda-f07d72ceab95" (UID: "c2cc0d64-e3d1-429e-8eda-f07d72ceab95"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:17:11.887228 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.887213    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "c2cc0d64-e3d1-429e-8eda-f07d72ceab95" (UID: "c2cc0d64-e3d1-429e-8eda-f07d72ceab95"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:11.887759 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.887737    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "c2cc0d64-e3d1-429e-8eda-f07d72ceab95" (UID: "c2cc0d64-e3d1-429e-8eda-f07d72ceab95"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:11.890954 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.890930    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "c2cc0d64-e3d1-429e-8eda-f07d72ceab95" (UID: "c2cc0d64-e3d1-429e-8eda-f07d72ceab95"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:11.898258 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.898224    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-web-config" (OuterVolumeSpecName: "web-config") pod "c2cc0d64-e3d1-429e-8eda-f07d72ceab95" (UID: "c2cc0d64-e3d1-429e-8eda-f07d72ceab95"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:11.984780 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.984734    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:17:11.984780 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.984778    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:17:11.984780 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.984790    2583 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-tls-assets\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:17:11.984780 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.984799    2583 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-web-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:17:11.985030 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.984808    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-secret-alertmanager-main-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:17:11.985030 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.984818    2583 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-cluster-tls-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:17:11.985030 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.984826    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx9q8\" (UniqueName: \"kubernetes.io/projected/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-kube-api-access-qx9q8\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:17:11.985030 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.984834    2583 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-config-out\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:17:11.985030 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.984842    2583 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-config-volume\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:17:11.985030 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.984850    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:17:11.985030 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:11.984858    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2cc0d64-e3d1-429e-8eda-f07d72ceab95-metrics-client-ca\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:17:12.104590 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.104548    2583 generic.go:356] "Generic (PLEG): container finished" podID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" containerID="893620c671ff83926e981be6776e2888269b028d1f37836eb5dc4a7da1cde2bd" exitCode=0
Jul 02 13:17:12.104590 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.104581    2583 generic.go:356] "Generic (PLEG): container finished" podID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" containerID="ada7017e79d620eb721290bfbb767d8f1ad3f37ffb4ca5e041322e33e2401d50" exitCode=0
Jul 02 13:17:12.104773 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.104616    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c2cc0d64-e3d1-429e-8eda-f07d72ceab95","Type":"ContainerDied","Data":"893620c671ff83926e981be6776e2888269b028d1f37836eb5dc4a7da1cde2bd"}
Jul 02 13:17:12.104773 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.104636    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.104773 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.104691    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c2cc0d64-e3d1-429e-8eda-f07d72ceab95","Type":"ContainerDied","Data":"ada7017e79d620eb721290bfbb767d8f1ad3f37ffb4ca5e041322e33e2401d50"}
Jul 02 13:17:12.104773 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.104734    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c2cc0d64-e3d1-429e-8eda-f07d72ceab95","Type":"ContainerDied","Data":"1f91c0a76f0abf1c957e9f9c4b728ebe2ee966c6ab2f08912af49229c6d24433"}
Jul 02 13:17:12.104970 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.104772    2583 scope.go:117] "RemoveContainer" containerID="097db36fa3ba4890896b738e466d6f28891071b2ef3b656c0b7f069f08751d47"
Jul 02 13:17:12.113257 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.113237    2583 scope.go:117] "RemoveContainer" containerID="893620c671ff83926e981be6776e2888269b028d1f37836eb5dc4a7da1cde2bd"
Jul 02 13:17:12.120316 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.120299    2583 scope.go:117] "RemoveContainer" containerID="09c787349750c7e5a9bd9a795dbee3261e5c3ae3b7dde349cd937e622da77084"
Jul 02 13:17:12.127575 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.127556    2583 scope.go:117] "RemoveContainer" containerID="ada7017e79d620eb721290bfbb767d8f1ad3f37ffb4ca5e041322e33e2401d50"
Jul 02 13:17:12.132003 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.131978    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 13:17:12.137129 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.137106    2583 scope.go:117] "RemoveContainer" containerID="b355077d48fb0a743a04f5bd919565534855f61be8801b965a697754b2cb9a20"
Jul 02 13:17:12.137541 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.137521    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 13:17:12.144685 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.144662    2583 scope.go:117] "RemoveContainer" containerID="b79d203129c1d4eacd6b84b1c22adb9619997a37836a1e01551d22d25007c1e9"
Jul 02 13:17:12.152284 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.152265    2583 scope.go:117] "RemoveContainer" containerID="2bade4f1d8a8dd313d3ca7dfc3553c3832b8c32c695c43aaf864cebfb205bba4"
Jul 02 13:17:12.159729 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.159710    2583 scope.go:117] "RemoveContainer" containerID="097db36fa3ba4890896b738e466d6f28891071b2ef3b656c0b7f069f08751d47"
Jul 02 13:17:12.160047 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:17:12.160024    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"097db36fa3ba4890896b738e466d6f28891071b2ef3b656c0b7f069f08751d47\": container with ID starting with 097db36fa3ba4890896b738e466d6f28891071b2ef3b656c0b7f069f08751d47 not found: ID does not exist" containerID="097db36fa3ba4890896b738e466d6f28891071b2ef3b656c0b7f069f08751d47"
Jul 02 13:17:12.160114 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.160057    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"097db36fa3ba4890896b738e466d6f28891071b2ef3b656c0b7f069f08751d47"} err="failed to get container status \"097db36fa3ba4890896b738e466d6f28891071b2ef3b656c0b7f069f08751d47\": rpc error: code = NotFound desc = could not find container \"097db36fa3ba4890896b738e466d6f28891071b2ef3b656c0b7f069f08751d47\": container with ID starting with 097db36fa3ba4890896b738e466d6f28891071b2ef3b656c0b7f069f08751d47 not found: ID does not exist"
Jul 02 13:17:12.160114 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.160091    2583 scope.go:117] "RemoveContainer" containerID="893620c671ff83926e981be6776e2888269b028d1f37836eb5dc4a7da1cde2bd"
Jul 02 13:17:12.160353 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:17:12.160336    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"893620c671ff83926e981be6776e2888269b028d1f37836eb5dc4a7da1cde2bd\": container with ID starting with 893620c671ff83926e981be6776e2888269b028d1f37836eb5dc4a7da1cde2bd not found: ID does not exist" containerID="893620c671ff83926e981be6776e2888269b028d1f37836eb5dc4a7da1cde2bd"
Jul 02 13:17:12.160407 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.160360    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"893620c671ff83926e981be6776e2888269b028d1f37836eb5dc4a7da1cde2bd"} err="failed to get container status \"893620c671ff83926e981be6776e2888269b028d1f37836eb5dc4a7da1cde2bd\": rpc error: code = NotFound desc = could not find container \"893620c671ff83926e981be6776e2888269b028d1f37836eb5dc4a7da1cde2bd\": container with ID starting with 893620c671ff83926e981be6776e2888269b028d1f37836eb5dc4a7da1cde2bd not found: ID does not exist"
Jul 02 13:17:12.160407 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.160375    2583 scope.go:117] "RemoveContainer" containerID="09c787349750c7e5a9bd9a795dbee3261e5c3ae3b7dde349cd937e622da77084"
Jul 02 13:17:12.160612 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:17:12.160592    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09c787349750c7e5a9bd9a795dbee3261e5c3ae3b7dde349cd937e622da77084\": container with ID starting with 09c787349750c7e5a9bd9a795dbee3261e5c3ae3b7dde349cd937e622da77084 not found: ID does not exist" containerID="09c787349750c7e5a9bd9a795dbee3261e5c3ae3b7dde349cd937e622da77084"
Jul 02 13:17:12.160671 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.160618    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09c787349750c7e5a9bd9a795dbee3261e5c3ae3b7dde349cd937e622da77084"} err="failed to get container status \"09c787349750c7e5a9bd9a795dbee3261e5c3ae3b7dde349cd937e622da77084\": rpc error: code = NotFound desc = could not find container \"09c787349750c7e5a9bd9a795dbee3261e5c3ae3b7dde349cd937e622da77084\": container with ID starting with 09c787349750c7e5a9bd9a795dbee3261e5c3ae3b7dde349cd937e622da77084 not found: ID does not exist"
Jul 02 13:17:12.160671 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.160632    2583 scope.go:117] "RemoveContainer" containerID="ada7017e79d620eb721290bfbb767d8f1ad3f37ffb4ca5e041322e33e2401d50"
Jul 02 13:17:12.160862 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:17:12.160845    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ada7017e79d620eb721290bfbb767d8f1ad3f37ffb4ca5e041322e33e2401d50\": container with ID starting with ada7017e79d620eb721290bfbb767d8f1ad3f37ffb4ca5e041322e33e2401d50 not found: ID does not exist" containerID="ada7017e79d620eb721290bfbb767d8f1ad3f37ffb4ca5e041322e33e2401d50"
Jul 02 13:17:12.160914 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.160864    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ada7017e79d620eb721290bfbb767d8f1ad3f37ffb4ca5e041322e33e2401d50"} err="failed to get container status \"ada7017e79d620eb721290bfbb767d8f1ad3f37ffb4ca5e041322e33e2401d50\": rpc error: code = NotFound desc = could not find container \"ada7017e79d620eb721290bfbb767d8f1ad3f37ffb4ca5e041322e33e2401d50\": container with ID starting with ada7017e79d620eb721290bfbb767d8f1ad3f37ffb4ca5e041322e33e2401d50 not found: ID does not exist"
Jul 02 13:17:12.160914 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.160876    2583 scope.go:117] "RemoveContainer" containerID="b355077d48fb0a743a04f5bd919565534855f61be8801b965a697754b2cb9a20"
Jul 02 13:17:12.161127 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:17:12.161103    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b355077d48fb0a743a04f5bd919565534855f61be8801b965a697754b2cb9a20\": container with ID starting with b355077d48fb0a743a04f5bd919565534855f61be8801b965a697754b2cb9a20 not found: ID does not exist" containerID="b355077d48fb0a743a04f5bd919565534855f61be8801b965a697754b2cb9a20"
Jul 02 13:17:12.161234 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.161131    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b355077d48fb0a743a04f5bd919565534855f61be8801b965a697754b2cb9a20"} err="failed to get container status \"b355077d48fb0a743a04f5bd919565534855f61be8801b965a697754b2cb9a20\": rpc error: code = NotFound desc = could not find container \"b355077d48fb0a743a04f5bd919565534855f61be8801b965a697754b2cb9a20\": container with ID starting with b355077d48fb0a743a04f5bd919565534855f61be8801b965a697754b2cb9a20 not found: ID does not exist"
Jul 02 13:17:12.161234 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.161154    2583 scope.go:117] "RemoveContainer" containerID="b79d203129c1d4eacd6b84b1c22adb9619997a37836a1e01551d22d25007c1e9"
Jul 02 13:17:12.161492 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:17:12.161466    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b79d203129c1d4eacd6b84b1c22adb9619997a37836a1e01551d22d25007c1e9\": container with ID starting with b79d203129c1d4eacd6b84b1c22adb9619997a37836a1e01551d22d25007c1e9 not found: ID does not exist" containerID="b79d203129c1d4eacd6b84b1c22adb9619997a37836a1e01551d22d25007c1e9"
Jul 02 13:17:12.161619 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.161497    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b79d203129c1d4eacd6b84b1c22adb9619997a37836a1e01551d22d25007c1e9"} err="failed to get container status \"b79d203129c1d4eacd6b84b1c22adb9619997a37836a1e01551d22d25007c1e9\": rpc error: code = NotFound desc = could not find container \"b79d203129c1d4eacd6b84b1c22adb9619997a37836a1e01551d22d25007c1e9\": container with ID starting with b79d203129c1d4eacd6b84b1c22adb9619997a37836a1e01551d22d25007c1e9 not found: ID does not exist"
Jul 02 13:17:12.161619 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.161515    2583 scope.go:117] "RemoveContainer" containerID="2bade4f1d8a8dd313d3ca7dfc3553c3832b8c32c695c43aaf864cebfb205bba4"
Jul 02 13:17:12.161923 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:17:12.161896    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bade4f1d8a8dd313d3ca7dfc3553c3832b8c32c695c43aaf864cebfb205bba4\": container with ID starting with 2bade4f1d8a8dd313d3ca7dfc3553c3832b8c32c695c43aaf864cebfb205bba4 not found: ID does not exist" containerID="2bade4f1d8a8dd313d3ca7dfc3553c3832b8c32c695c43aaf864cebfb205bba4"
Jul 02 13:17:12.162006 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.161917    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bade4f1d8a8dd313d3ca7dfc3553c3832b8c32c695c43aaf864cebfb205bba4"} err="failed to get container status \"2bade4f1d8a8dd313d3ca7dfc3553c3832b8c32c695c43aaf864cebfb205bba4\": rpc error: code = NotFound desc = could not find container \"2bade4f1d8a8dd313d3ca7dfc3553c3832b8c32c695c43aaf864cebfb205bba4\": container with ID starting with 2bade4f1d8a8dd313d3ca7dfc3553c3832b8c32c695c43aaf864cebfb205bba4 not found: ID does not exist"
Jul 02 13:17:12.162006 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.161936    2583 scope.go:117] "RemoveContainer" containerID="097db36fa3ba4890896b738e466d6f28891071b2ef3b656c0b7f069f08751d47"
Jul 02 13:17:12.162191 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.162163    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"097db36fa3ba4890896b738e466d6f28891071b2ef3b656c0b7f069f08751d47"} err="failed to get container status \"097db36fa3ba4890896b738e466d6f28891071b2ef3b656c0b7f069f08751d47\": rpc error: code = NotFound desc = could not find container \"097db36fa3ba4890896b738e466d6f28891071b2ef3b656c0b7f069f08751d47\": container with ID starting with 097db36fa3ba4890896b738e466d6f28891071b2ef3b656c0b7f069f08751d47 not found: ID does not exist"
Jul 02 13:17:12.162250 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.162192    2583 scope.go:117] "RemoveContainer" containerID="893620c671ff83926e981be6776e2888269b028d1f37836eb5dc4a7da1cde2bd"
Jul 02 13:17:12.162440 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.162424    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"893620c671ff83926e981be6776e2888269b028d1f37836eb5dc4a7da1cde2bd"} err="failed to get container status \"893620c671ff83926e981be6776e2888269b028d1f37836eb5dc4a7da1cde2bd\": rpc error: code = NotFound desc = could not find container \"893620c671ff83926e981be6776e2888269b028d1f37836eb5dc4a7da1cde2bd\": container with ID starting with 893620c671ff83926e981be6776e2888269b028d1f37836eb5dc4a7da1cde2bd not found: ID does not exist"
Jul 02 13:17:12.162498 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.162440    2583 scope.go:117] "RemoveContainer" containerID="09c787349750c7e5a9bd9a795dbee3261e5c3ae3b7dde349cd937e622da77084"
Jul 02 13:17:12.162662 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.162646    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09c787349750c7e5a9bd9a795dbee3261e5c3ae3b7dde349cd937e622da77084"} err="failed to get container status \"09c787349750c7e5a9bd9a795dbee3261e5c3ae3b7dde349cd937e622da77084\": rpc error: code = NotFound desc = could not find container \"09c787349750c7e5a9bd9a795dbee3261e5c3ae3b7dde349cd937e622da77084\": container with ID starting with 09c787349750c7e5a9bd9a795dbee3261e5c3ae3b7dde349cd937e622da77084 not found: ID does not exist"
Jul 02 13:17:12.162662 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.162661    2583 scope.go:117] "RemoveContainer" containerID="ada7017e79d620eb721290bfbb767d8f1ad3f37ffb4ca5e041322e33e2401d50"
Jul 02 13:17:12.162858 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.162842    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ada7017e79d620eb721290bfbb767d8f1ad3f37ffb4ca5e041322e33e2401d50"} err="failed to get container status \"ada7017e79d620eb721290bfbb767d8f1ad3f37ffb4ca5e041322e33e2401d50\": rpc error: code = NotFound desc = could not find container \"ada7017e79d620eb721290bfbb767d8f1ad3f37ffb4ca5e041322e33e2401d50\": container with ID starting with ada7017e79d620eb721290bfbb767d8f1ad3f37ffb4ca5e041322e33e2401d50 not found: ID does not exist"
Jul 02 13:17:12.162898 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.162858    2583 scope.go:117] "RemoveContainer" containerID="b355077d48fb0a743a04f5bd919565534855f61be8801b965a697754b2cb9a20"
Jul 02 13:17:12.163057 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.163039    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b355077d48fb0a743a04f5bd919565534855f61be8801b965a697754b2cb9a20"} err="failed to get container status \"b355077d48fb0a743a04f5bd919565534855f61be8801b965a697754b2cb9a20\": rpc error: code = NotFound desc = could not find container \"b355077d48fb0a743a04f5bd919565534855f61be8801b965a697754b2cb9a20\": container with ID starting with b355077d48fb0a743a04f5bd919565534855f61be8801b965a697754b2cb9a20 not found: ID does not exist"
Jul 02 13:17:12.163105 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.163056    2583 scope.go:117] "RemoveContainer" containerID="b79d203129c1d4eacd6b84b1c22adb9619997a37836a1e01551d22d25007c1e9"
Jul 02 13:17:12.163258 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.163244    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b79d203129c1d4eacd6b84b1c22adb9619997a37836a1e01551d22d25007c1e9"} err="failed to get container status \"b79d203129c1d4eacd6b84b1c22adb9619997a37836a1e01551d22d25007c1e9\": rpc error: code = NotFound desc = could not find container \"b79d203129c1d4eacd6b84b1c22adb9619997a37836a1e01551d22d25007c1e9\": container with ID starting with b79d203129c1d4eacd6b84b1c22adb9619997a37836a1e01551d22d25007c1e9 not found: ID does not exist"
Jul 02 13:17:12.163296 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.163258    2583 scope.go:117] "RemoveContainer" containerID="2bade4f1d8a8dd313d3ca7dfc3553c3832b8c32c695c43aaf864cebfb205bba4"
Jul 02 13:17:12.163505 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.163481    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bade4f1d8a8dd313d3ca7dfc3553c3832b8c32c695c43aaf864cebfb205bba4"} err="failed to get container status \"2bade4f1d8a8dd313d3ca7dfc3553c3832b8c32c695c43aaf864cebfb205bba4\": rpc error: code = NotFound desc = could not find container \"2bade4f1d8a8dd313d3ca7dfc3553c3832b8c32c695c43aaf864cebfb205bba4\": container with ID starting with 2bade4f1d8a8dd313d3ca7dfc3553c3832b8c32c695c43aaf864cebfb205bba4 not found: ID does not exist"
Jul 02 13:17:12.174274 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.174250    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 13:17:12.174579 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.174565    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" containerName="config-reloader"
Jul 02 13:17:12.174625 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.174581    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" containerName="config-reloader"
Jul 02 13:17:12.174625 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.174593    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" containerName="init-config-reloader"
Jul 02 13:17:12.174625 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.174598    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" containerName="init-config-reloader"
Jul 02 13:17:12.174625 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.174606    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" containerName="kube-rbac-proxy-web"
Jul 02 13:17:12.174625 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.174611    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" containerName="kube-rbac-proxy-web"
Jul 02 13:17:12.174625 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.174618    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" containerName="kube-rbac-proxy-metric"
Jul 02 13:17:12.174625 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.174623    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" containerName="kube-rbac-proxy-metric"
Jul 02 13:17:12.174816 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.174630    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" containerName="kube-rbac-proxy"
Jul 02 13:17:12.174816 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.174635    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" containerName="kube-rbac-proxy"
Jul 02 13:17:12.174816 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.174646    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" containerName="prom-label-proxy"
Jul 02 13:17:12.174816 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.174651    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" containerName="prom-label-proxy"
Jul 02 13:17:12.174816 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.174664    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" containerName="alertmanager"
Jul 02 13:17:12.174816 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.174669    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" containerName="alertmanager"
Jul 02 13:17:12.174816 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.174714    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" containerName="kube-rbac-proxy-web"
Jul 02 13:17:12.174816 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.174724    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" containerName="alertmanager"
Jul 02 13:17:12.174816 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.174732    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" containerName="kube-rbac-proxy"
Jul 02 13:17:12.174816 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.174737    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" containerName="kube-rbac-proxy-metric"
Jul 02 13:17:12.174816 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.174743    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" containerName="config-reloader"
Jul 02 13:17:12.174816 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.174748    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" containerName="prom-label-proxy"
Jul 02 13:17:12.179082 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.179063    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.182551 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.182521    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 02 13:17:12.182657 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.182627    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 02 13:17:12.182753 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.182733    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-qqtmd\""
Jul 02 13:17:12.183024 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.183009    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 02 13:17:12.183087 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.183061    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 02 13:17:12.183217 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.183204    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 02 13:17:12.183400 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.183374    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 02 13:17:12.183400 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.183379    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 02 13:17:12.183694 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.183677    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 02 13:17:12.189963 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.189943    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 02 13:17:12.200072 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.200047    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 13:17:12.287398 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.287360    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5a6c204f-4d22-4e76-982e-98dcb22daa59-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.287650 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.287428    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a6c204f-4d22-4e76-982e-98dcb22daa59-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.287650 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.287502    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a6c204f-4d22-4e76-982e-98dcb22daa59-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.287650 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.287531    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a6c204f-4d22-4e76-982e-98dcb22daa59-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.287650 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.287556    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a6c204f-4d22-4e76-982e-98dcb22daa59-web-config\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.287650 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.287622    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5a6c204f-4d22-4e76-982e-98dcb22daa59-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.287650 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.287654    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a6c204f-4d22-4e76-982e-98dcb22daa59-config-out\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.287883 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.287681    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5a6c204f-4d22-4e76-982e-98dcb22daa59-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.287883 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.287701    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a6c204f-4d22-4e76-982e-98dcb22daa59-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.287883 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.287717    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j4tx\" (UniqueName: \"kubernetes.io/projected/5a6c204f-4d22-4e76-982e-98dcb22daa59-kube-api-access-4j4tx\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.287883 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.287745    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a6c204f-4d22-4e76-982e-98dcb22daa59-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.287883 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.287764    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5a6c204f-4d22-4e76-982e-98dcb22daa59-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.287883 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.287784    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5a6c204f-4d22-4e76-982e-98dcb22daa59-config-volume\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.388614 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.388507    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5a6c204f-4d22-4e76-982e-98dcb22daa59-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.388614 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.388558    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a6c204f-4d22-4e76-982e-98dcb22daa59-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.388614 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.388574    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j4tx\" (UniqueName: \"kubernetes.io/projected/5a6c204f-4d22-4e76-982e-98dcb22daa59-kube-api-access-4j4tx\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.388902 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.388681    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a6c204f-4d22-4e76-982e-98dcb22daa59-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.388902 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.388710    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5a6c204f-4d22-4e76-982e-98dcb22daa59-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.388902 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.388734    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5a6c204f-4d22-4e76-982e-98dcb22daa59-config-volume\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.388902 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.388798    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5a6c204f-4d22-4e76-982e-98dcb22daa59-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.388902 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.388826    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a6c204f-4d22-4e76-982e-98dcb22daa59-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.388902 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.388872    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a6c204f-4d22-4e76-982e-98dcb22daa59-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.388902 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.388901    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a6c204f-4d22-4e76-982e-98dcb22daa59-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.389233 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.388932    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a6c204f-4d22-4e76-982e-98dcb22daa59-web-config\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.389233 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.388984    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5a6c204f-4d22-4e76-982e-98dcb22daa59-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.389233 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.388991    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5a6c204f-4d22-4e76-982e-98dcb22daa59-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.389233 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.389046    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a6c204f-4d22-4e76-982e-98dcb22daa59-config-out\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.391627 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.391598    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a6c204f-4d22-4e76-982e-98dcb22daa59-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.391752 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.391688    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a6c204f-4d22-4e76-982e-98dcb22daa59-config-out\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.391752 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.391702    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5a6c204f-4d22-4e76-982e-98dcb22daa59-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.391752 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.391735    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a6c204f-4d22-4e76-982e-98dcb22daa59-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.391926 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.391821    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5a6c204f-4d22-4e76-982e-98dcb22daa59-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.391926 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.391841    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5a6c204f-4d22-4e76-982e-98dcb22daa59-config-volume\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.392023 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.391979    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a6c204f-4d22-4e76-982e-98dcb22daa59-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.392465 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.392442    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5a6c204f-4d22-4e76-982e-98dcb22daa59-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.392554 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.392538    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a6c204f-4d22-4e76-982e-98dcb22daa59-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.392799 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.392772    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a6c204f-4d22-4e76-982e-98dcb22daa59-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.394024 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.393997    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a6c204f-4d22-4e76-982e-98dcb22daa59-web-config\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.400682 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.400661    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j4tx\" (UniqueName: \"kubernetes.io/projected/5a6c204f-4d22-4e76-982e-98dcb22daa59-kube-api-access-4j4tx\") pod \"alertmanager-main-0\" (UID: \"5a6c204f-4d22-4e76-982e-98dcb22daa59\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.489766 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.489734    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 02 13:17:12.652623 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.652599    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 02 13:17:12.655082 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:17:12.655055    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a6c204f_4d22_4e76_982e_98dcb22daa59.slice/crio-0995b6547b31e8b7abb5676f1d65a344045e71edc4432ee99217d652c0e9846a WatchSource:0}: Error finding container 0995b6547b31e8b7abb5676f1d65a344045e71edc4432ee99217d652c0e9846a: Status 404 returned error can't find the container with id 0995b6547b31e8b7abb5676f1d65a344045e71edc4432ee99217d652c0e9846a
Jul 02 13:17:12.669969 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.669942    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:12.770085 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:12.770047    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:13.113051 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:13.112967    2583 generic.go:356] "Generic (PLEG): container finished" podID="5a6c204f-4d22-4e76-982e-98dcb22daa59" containerID="808c53e0bc149f802240aceaadc2083a6c36adbc17d15fdd7577cb1e09637d21" exitCode=0
Jul 02 13:17:13.113051 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:13.113030    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a6c204f-4d22-4e76-982e-98dcb22daa59","Type":"ContainerDied","Data":"808c53e0bc149f802240aceaadc2083a6c36adbc17d15fdd7577cb1e09637d21"}
Jul 02 13:17:13.113463 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:13.113080    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a6c204f-4d22-4e76-982e-98dcb22daa59","Type":"ContainerStarted","Data":"0995b6547b31e8b7abb5676f1d65a344045e71edc4432ee99217d652c0e9846a"}
Jul 02 13:17:13.232856 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:13.232828    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2cc0d64-e3d1-429e-8eda-f07d72ceab95" path="/var/lib/kubelet/pods/c2cc0d64-e3d1-429e-8eda-f07d72ceab95/volumes"
Jul 02 13:17:14.119608 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:14.119567    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a6c204f-4d22-4e76-982e-98dcb22daa59","Type":"ContainerStarted","Data":"e5d4678a60d012ede469ec5365d0a9fe7d0ad5045f6c597272a249a3b1fc5f79"}
Jul 02 13:17:14.119983 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:14.119684    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a6c204f-4d22-4e76-982e-98dcb22daa59","Type":"ContainerStarted","Data":"78d14144fa91e9384339132a9c2417351a4d02680e8e5f394dfec3f83e3acb9b"}
Jul 02 13:17:14.119983 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:14.119728    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a6c204f-4d22-4e76-982e-98dcb22daa59","Type":"ContainerStarted","Data":"5df940242e3914722d4f6ff6257219dd56a54207c9298c7cd5bfa73f0e3bca3f"}
Jul 02 13:17:14.119983 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:14.119762    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a6c204f-4d22-4e76-982e-98dcb22daa59","Type":"ContainerStarted","Data":"88ec1275bed1bf07e2f4b08b2d71f462662731fbab14b690171ae5b41c1a82f0"}
Jul 02 13:17:14.119983 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:14.119800    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a6c204f-4d22-4e76-982e-98dcb22daa59","Type":"ContainerStarted","Data":"07125ec130ce115ecfaa7f567fabbf0cb4205d3bfb03af6911b8bac1b901bd02"}
Jul 02 13:17:14.119983 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:14.119835    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a6c204f-4d22-4e76-982e-98dcb22daa59","Type":"ContainerStarted","Data":"43ac979fda10cc772b262ba57efed5a03be4b33663abb6d127b6cd3c18acd539"}
Jul 02 13:17:14.150490 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:14.150420    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.134660511 podStartE2EDuration="2.150373489s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.712978ms" isStatefulPod=false podCreationTimestamp="2026-07-02 13:17:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:17:14.149773349 +0000 UTC m=+145.377684527" watchObservedRunningTime="2026-07-02 13:17:14.150373489 +0000 UTC m=+145.378284665"
Jul 02 13:17:14.861604 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:14.861545    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 13:17:14.862072 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:14.862036    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5772ab56-d521-4af5-8c02-4d017569eb98" containerName="kube-rbac-proxy-thanos" containerID="cri-o://69dbdde8164c3e3c5eb3946a35e9f6dca70b2785162bb3498bcb7f4fb78a7082" gracePeriod=600
Jul 02 13:17:14.862156 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:14.862051    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5772ab56-d521-4af5-8c02-4d017569eb98" containerName="kube-rbac-proxy-web" containerID="cri-o://c7d3dadeb32b528bae0c69d223f0ed263240d57a73e6df86a17e331733bddf60" gracePeriod=600
Jul 02 13:17:14.862434 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:14.862208    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5772ab56-d521-4af5-8c02-4d017569eb98" containerName="kube-rbac-proxy" containerID="cri-o://76c8f8c577ab675b334167508900629fa74ac333a5e4e3a003d80938b77bc813" gracePeriod=600
Jul 02 13:17:14.862434 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:14.862315    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5772ab56-d521-4af5-8c02-4d017569eb98" containerName="thanos-sidecar" containerID="cri-o://1c586f4447e899f63eb93699c7377ca8579b1ab5e2f9466f4babbfb0bb1275b4" gracePeriod=600
Jul 02 13:17:14.862434 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:14.862432    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5772ab56-d521-4af5-8c02-4d017569eb98" containerName="config-reloader" containerID="cri-o://580d6f6bb79535caffffba94bf5b4ddeca1c77f84377f47815d14cd12f362eee" gracePeriod=600
Jul 02 13:17:14.862740 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:14.862597    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5772ab56-d521-4af5-8c02-4d017569eb98" containerName="prometheus" containerID="cri-o://b411e9a05b50c7209cbf1d2c1f8f438ec6e4457401b00ae96a90f2687c2d91d7" gracePeriod=600
Jul 02 13:17:15.128882 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.128851    2583 generic.go:356] "Generic (PLEG): container finished" podID="5772ab56-d521-4af5-8c02-4d017569eb98" containerID="69dbdde8164c3e3c5eb3946a35e9f6dca70b2785162bb3498bcb7f4fb78a7082" exitCode=0
Jul 02 13:17:15.128882 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.128875    2583 generic.go:356] "Generic (PLEG): container finished" podID="5772ab56-d521-4af5-8c02-4d017569eb98" containerID="76c8f8c577ab675b334167508900629fa74ac333a5e4e3a003d80938b77bc813" exitCode=0
Jul 02 13:17:15.128882 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.128884    2583 generic.go:356] "Generic (PLEG): container finished" podID="5772ab56-d521-4af5-8c02-4d017569eb98" containerID="c7d3dadeb32b528bae0c69d223f0ed263240d57a73e6df86a17e331733bddf60" exitCode=0
Jul 02 13:17:15.129294 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.128890    2583 generic.go:356] "Generic (PLEG): container finished" podID="5772ab56-d521-4af5-8c02-4d017569eb98" containerID="1c586f4447e899f63eb93699c7377ca8579b1ab5e2f9466f4babbfb0bb1275b4" exitCode=0
Jul 02 13:17:15.129294 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.128899    2583 generic.go:356] "Generic (PLEG): container finished" podID="5772ab56-d521-4af5-8c02-4d017569eb98" containerID="580d6f6bb79535caffffba94bf5b4ddeca1c77f84377f47815d14cd12f362eee" exitCode=0
Jul 02 13:17:15.129294 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.128907    2583 generic.go:356] "Generic (PLEG): container finished" podID="5772ab56-d521-4af5-8c02-4d017569eb98" containerID="b411e9a05b50c7209cbf1d2c1f8f438ec6e4457401b00ae96a90f2687c2d91d7" exitCode=0
Jul 02 13:17:15.129294 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.128923    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5772ab56-d521-4af5-8c02-4d017569eb98","Type":"ContainerDied","Data":"69dbdde8164c3e3c5eb3946a35e9f6dca70b2785162bb3498bcb7f4fb78a7082"}
Jul 02 13:17:15.129294 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.128999    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5772ab56-d521-4af5-8c02-4d017569eb98","Type":"ContainerDied","Data":"76c8f8c577ab675b334167508900629fa74ac333a5e4e3a003d80938b77bc813"}
Jul 02 13:17:15.129294 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.129041    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5772ab56-d521-4af5-8c02-4d017569eb98","Type":"ContainerDied","Data":"c7d3dadeb32b528bae0c69d223f0ed263240d57a73e6df86a17e331733bddf60"}
Jul 02 13:17:15.129294 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.129078    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5772ab56-d521-4af5-8c02-4d017569eb98","Type":"ContainerDied","Data":"1c586f4447e899f63eb93699c7377ca8579b1ab5e2f9466f4babbfb0bb1275b4"}
Jul 02 13:17:15.129294 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.129119    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5772ab56-d521-4af5-8c02-4d017569eb98","Type":"ContainerDied","Data":"580d6f6bb79535caffffba94bf5b4ddeca1c77f84377f47815d14cd12f362eee"}
Jul 02 13:17:15.129294 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.129158    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5772ab56-d521-4af5-8c02-4d017569eb98","Type":"ContainerDied","Data":"b411e9a05b50c7209cbf1d2c1f8f438ec6e4457401b00ae96a90f2687c2d91d7"}
Jul 02 13:17:15.129294 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.129197    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5772ab56-d521-4af5-8c02-4d017569eb98","Type":"ContainerDied","Data":"3c595b7396770c74d5966520f7ca141c075a8410d71604e9ef2bc91ccae3b539"}
Jul 02 13:17:15.129294 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.129237    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c595b7396770c74d5966520f7ca141c075a8410d71604e9ef2bc91ccae3b539"
Jul 02 13:17:15.130014 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.129992    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:15.214983 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.214943    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"5772ab56-d521-4af5-8c02-4d017569eb98\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") "
Jul 02 13:17:15.214983 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.214987    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-config\") pod \"5772ab56-d521-4af5-8c02-4d017569eb98\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") "
Jul 02 13:17:15.215286 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.215015    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-configmap-kubelet-serving-ca-bundle\") pod \"5772ab56-d521-4af5-8c02-4d017569eb98\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") "
Jul 02 13:17:15.215286 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.215038    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-configmap-serving-certs-ca-bundle\") pod \"5772ab56-d521-4af5-8c02-4d017569eb98\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") "
Jul 02 13:17:15.215286 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.215066    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5772ab56-d521-4af5-8c02-4d017569eb98-tls-assets\") pod \"5772ab56-d521-4af5-8c02-4d017569eb98\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") "
Jul 02 13:17:15.215286 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.215102    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5772ab56-d521-4af5-8c02-4d017569eb98-prometheus-k8s-db\") pod \"5772ab56-d521-4af5-8c02-4d017569eb98\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") "
Jul 02 13:17:15.215286 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.215126    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-prometheus-k8s-rulefiles-2\") pod \"5772ab56-d521-4af5-8c02-4d017569eb98\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") "
Jul 02 13:17:15.215286 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.215158    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-thanos-prometheus-http-client-file\") pod \"5772ab56-d521-4af5-8c02-4d017569eb98\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") "
Jul 02 13:17:15.215286 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.215186    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-configmap-metrics-client-ca\") pod \"5772ab56-d521-4af5-8c02-4d017569eb98\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") "
Jul 02 13:17:15.215286 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.215210    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhxkg\" (UniqueName: \"kubernetes.io/projected/5772ab56-d521-4af5-8c02-4d017569eb98-kube-api-access-vhxkg\") pod \"5772ab56-d521-4af5-8c02-4d017569eb98\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") "
Jul 02 13:17:15.215286 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.215243    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"5772ab56-d521-4af5-8c02-4d017569eb98\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") "
Jul 02 13:17:15.215286 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.215273    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-web-config\") pod \"5772ab56-d521-4af5-8c02-4d017569eb98\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") "
Jul 02 13:17:15.215783 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.215335    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-prometheus-k8s-tls\") pod \"5772ab56-d521-4af5-8c02-4d017569eb98\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") "
Jul 02 13:17:15.215783 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.215408    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-prometheus-k8s-rulefiles-1\") pod \"5772ab56-d521-4af5-8c02-4d017569eb98\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") "
Jul 02 13:17:15.215783 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.215434    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-metrics-client-certs\") pod \"5772ab56-d521-4af5-8c02-4d017569eb98\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") "
Jul 02 13:17:15.215783 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.215444    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "5772ab56-d521-4af5-8c02-4d017569eb98" (UID: "5772ab56-d521-4af5-8c02-4d017569eb98"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:17:15.215783 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.215468    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5772ab56-d521-4af5-8c02-4d017569eb98-config-out\") pod \"5772ab56-d521-4af5-8c02-4d017569eb98\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") "
Jul 02 13:17:15.215783 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.215493    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-grpc-tls\") pod \"5772ab56-d521-4af5-8c02-4d017569eb98\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") "
Jul 02 13:17:15.215783 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.215529    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-prometheus-k8s-rulefiles-0\") pod \"5772ab56-d521-4af5-8c02-4d017569eb98\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") "
Jul 02 13:17:15.215783 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.215560    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-prometheus-trusted-ca-bundle\") pod \"5772ab56-d521-4af5-8c02-4d017569eb98\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") "
Jul 02 13:17:15.215783 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.215590    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-kube-rbac-proxy\") pod \"5772ab56-d521-4af5-8c02-4d017569eb98\" (UID: \"5772ab56-d521-4af5-8c02-4d017569eb98\") "
Jul 02 13:17:15.215783 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.215732    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "5772ab56-d521-4af5-8c02-4d017569eb98" (UID: "5772ab56-d521-4af5-8c02-4d017569eb98"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:17:15.216239 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.215863    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "5772ab56-d521-4af5-8c02-4d017569eb98" (UID: "5772ab56-d521-4af5-8c02-4d017569eb98"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:17:15.216239 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.216182    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5772ab56-d521-4af5-8c02-4d017569eb98-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "5772ab56-d521-4af5-8c02-4d017569eb98" (UID: "5772ab56-d521-4af5-8c02-4d017569eb98"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:17:15.216378 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.216270    2583 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:17:15.217508 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.217471    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "5772ab56-d521-4af5-8c02-4d017569eb98" (UID: "5772ab56-d521-4af5-8c02-4d017569eb98"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:17:15.217643 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.217522    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "5772ab56-d521-4af5-8c02-4d017569eb98" (UID: "5772ab56-d521-4af5-8c02-4d017569eb98"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:17:15.217643 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.217553    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:17:15.217643 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.217581    2583 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-configmap-metrics-client-ca\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:17:15.217884 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.217859    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-config" (OuterVolumeSpecName: "config") pod "5772ab56-d521-4af5-8c02-4d017569eb98" (UID: "5772ab56-d521-4af5-8c02-4d017569eb98"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:15.218202 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.218167    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "5772ab56-d521-4af5-8c02-4d017569eb98" (UID: "5772ab56-d521-4af5-8c02-4d017569eb98"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:17:15.218620 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.218588    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "5772ab56-d521-4af5-8c02-4d017569eb98" (UID: "5772ab56-d521-4af5-8c02-4d017569eb98"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:15.219450 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.219407    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "5772ab56-d521-4af5-8c02-4d017569eb98" (UID: "5772ab56-d521-4af5-8c02-4d017569eb98"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:17:15.219725 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.219691    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5772ab56-d521-4af5-8c02-4d017569eb98-kube-api-access-vhxkg" (OuterVolumeSpecName: "kube-api-access-vhxkg") pod "5772ab56-d521-4af5-8c02-4d017569eb98" (UID: "5772ab56-d521-4af5-8c02-4d017569eb98"). InnerVolumeSpecName "kube-api-access-vhxkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:17:15.219985 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.219923    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "5772ab56-d521-4af5-8c02-4d017569eb98" (UID: "5772ab56-d521-4af5-8c02-4d017569eb98"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:15.220423 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.220317    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "5772ab56-d521-4af5-8c02-4d017569eb98" (UID: "5772ab56-d521-4af5-8c02-4d017569eb98"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:15.220423 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.220359    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5772ab56-d521-4af5-8c02-4d017569eb98-config-out" (OuterVolumeSpecName: "config-out") pod "5772ab56-d521-4af5-8c02-4d017569eb98" (UID: "5772ab56-d521-4af5-8c02-4d017569eb98"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:17:15.220610 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.220552    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5772ab56-d521-4af5-8c02-4d017569eb98-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "5772ab56-d521-4af5-8c02-4d017569eb98" (UID: "5772ab56-d521-4af5-8c02-4d017569eb98"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:17:15.220885 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.220857    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "5772ab56-d521-4af5-8c02-4d017569eb98" (UID: "5772ab56-d521-4af5-8c02-4d017569eb98"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:15.221600 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.221017    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "5772ab56-d521-4af5-8c02-4d017569eb98" (UID: "5772ab56-d521-4af5-8c02-4d017569eb98"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:15.221600 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.221437    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "5772ab56-d521-4af5-8c02-4d017569eb98" (UID: "5772ab56-d521-4af5-8c02-4d017569eb98"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:15.221990 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.221971    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "5772ab56-d521-4af5-8c02-4d017569eb98" (UID: "5772ab56-d521-4af5-8c02-4d017569eb98"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:15.232467 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.232432    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-web-config" (OuterVolumeSpecName: "web-config") pod "5772ab56-d521-4af5-8c02-4d017569eb98" (UID: "5772ab56-d521-4af5-8c02-4d017569eb98"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:17:15.318849 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.318814    2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:17:15.318849 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.318843    2583 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:17:15.319073 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.318856    2583 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:17:15.319073 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.318870    2583 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5772ab56-d521-4af5-8c02-4d017569eb98-tls-assets\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:17:15.319073 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.318884    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5772ab56-d521-4af5-8c02-4d017569eb98-prometheus-k8s-db\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:17:15.319073 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.318895    2583 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:17:15.319073 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.318906    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhxkg\" (UniqueName: \"kubernetes.io/projected/5772ab56-d521-4af5-8c02-4d017569eb98-kube-api-access-vhxkg\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:17:15.319073 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.318920    2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:17:15.319073 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.318932    2583 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-web-config\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:17:15.319073 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.318943    2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:17:15.319073 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.318954    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:17:15.319073 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.318965    2583 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-metrics-client-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:17:15.319073 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.318979    2583 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5772ab56-d521-4af5-8c02-4d017569eb98-config-out\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:17:15.319073 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.318990    2583 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-grpc-tls\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:17:15.319073 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.319001    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:17:15.319073 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.319015    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5772ab56-d521-4af5-8c02-4d017569eb98-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:17:15.319073 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:15.319027    2583 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5772ab56-d521-4af5-8c02-4d017569eb98-secret-kube-rbac-proxy\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:17:16.133200 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.133105    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.160423 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.160374    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 13:17:16.166973 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.166946    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 13:17:16.194017 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.193981    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 13:17:16.194404 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.194365    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5772ab56-d521-4af5-8c02-4d017569eb98" containerName="config-reloader"
Jul 02 13:17:16.194404 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.194402    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5772ab56-d521-4af5-8c02-4d017569eb98" containerName="config-reloader"
Jul 02 13:17:16.194542 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.194416    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5772ab56-d521-4af5-8c02-4d017569eb98" containerName="kube-rbac-proxy-web"
Jul 02 13:17:16.194542 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.194425    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5772ab56-d521-4af5-8c02-4d017569eb98" containerName="kube-rbac-proxy-web"
Jul 02 13:17:16.194542 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.194438    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5772ab56-d521-4af5-8c02-4d017569eb98" containerName="kube-rbac-proxy-thanos"
Jul 02 13:17:16.194542 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.194446    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5772ab56-d521-4af5-8c02-4d017569eb98" containerName="kube-rbac-proxy-thanos"
Jul 02 13:17:16.194542 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.194464    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5772ab56-d521-4af5-8c02-4d017569eb98" containerName="prometheus"
Jul 02 13:17:16.194542 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.194472    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5772ab56-d521-4af5-8c02-4d017569eb98" containerName="prometheus"
Jul 02 13:17:16.194542 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.194480    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5772ab56-d521-4af5-8c02-4d017569eb98" containerName="thanos-sidecar"
Jul 02 13:17:16.194542 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.194488    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5772ab56-d521-4af5-8c02-4d017569eb98" containerName="thanos-sidecar"
Jul 02 13:17:16.194542 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.194502    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5772ab56-d521-4af5-8c02-4d017569eb98" containerName="kube-rbac-proxy"
Jul 02 13:17:16.194542 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.194511    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5772ab56-d521-4af5-8c02-4d017569eb98" containerName="kube-rbac-proxy"
Jul 02 13:17:16.194542 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.194522    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5772ab56-d521-4af5-8c02-4d017569eb98" containerName="init-config-reloader"
Jul 02 13:17:16.194542 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.194530    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5772ab56-d521-4af5-8c02-4d017569eb98" containerName="init-config-reloader"
Jul 02 13:17:16.195105 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.194609    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5772ab56-d521-4af5-8c02-4d017569eb98" containerName="kube-rbac-proxy"
Jul 02 13:17:16.195105 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.194623    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5772ab56-d521-4af5-8c02-4d017569eb98" containerName="kube-rbac-proxy-web"
Jul 02 13:17:16.195105 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.194635    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5772ab56-d521-4af5-8c02-4d017569eb98" containerName="kube-rbac-proxy-thanos"
Jul 02 13:17:16.195105 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.194648    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5772ab56-d521-4af5-8c02-4d017569eb98" containerName="thanos-sidecar"
Jul 02 13:17:16.195105 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.194657    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5772ab56-d521-4af5-8c02-4d017569eb98" containerName="prometheus"
Jul 02 13:17:16.195105 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.194666    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5772ab56-d521-4af5-8c02-4d017569eb98" containerName="config-reloader"
Jul 02 13:17:16.198495 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.198475    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.201312 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.201286    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 02 13:17:16.201462 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.201444    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jh8sh\""
Jul 02 13:17:16.201561 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.201482    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 02 13:17:16.201686 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.201667    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9m5jmmh8jgtj5\""
Jul 02 13:17:16.201813 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.201690    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 02 13:17:16.201813 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.201781    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 02 13:17:16.201925 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.201830    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 02 13:17:16.201982 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.201963    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 02 13:17:16.202038 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.201977    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 02 13:17:16.202038 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.202005    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 02 13:17:16.202038 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.202030    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 02 13:17:16.202187 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.201968    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 02 13:17:16.202470 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.202452    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 02 13:17:16.202565 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.202531    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 02 13:17:16.206595 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.206553    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 02 13:17:16.208737 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.208721    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 02 13:17:16.216633 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.216609    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 13:17:16.327188 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.327147    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/965caecb-d1eb-45a2-a48a-1495330553f8-web-config\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.327188 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.327198    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/965caecb-d1eb-45a2-a48a-1495330553f8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.327482 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.327217    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/965caecb-d1eb-45a2-a48a-1495330553f8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.327482 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.327273    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/965caecb-d1eb-45a2-a48a-1495330553f8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.327482 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.327310    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlqk8\" (UniqueName: \"kubernetes.io/projected/965caecb-d1eb-45a2-a48a-1495330553f8-kube-api-access-nlqk8\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.327482 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.327351    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/965caecb-d1eb-45a2-a48a-1495330553f8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.327482 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.327414    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/965caecb-d1eb-45a2-a48a-1495330553f8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.327482 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.327438    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/965caecb-d1eb-45a2-a48a-1495330553f8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.327482 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.327471    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/965caecb-d1eb-45a2-a48a-1495330553f8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.327831 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.327519    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/965caecb-d1eb-45a2-a48a-1495330553f8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.327831 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.327542    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/965caecb-d1eb-45a2-a48a-1495330553f8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.327831 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.327574    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/965caecb-d1eb-45a2-a48a-1495330553f8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.327831 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.327614    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/965caecb-d1eb-45a2-a48a-1495330553f8-config-out\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.327831 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.327665    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/965caecb-d1eb-45a2-a48a-1495330553f8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.327831 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.327692    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/965caecb-d1eb-45a2-a48a-1495330553f8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.327831 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.327718    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/965caecb-d1eb-45a2-a48a-1495330553f8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.327831 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.327774    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/965caecb-d1eb-45a2-a48a-1495330553f8-config\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.327831 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.327801    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/965caecb-d1eb-45a2-a48a-1495330553f8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.328140 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.327836    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/965caecb-d1eb-45a2-a48a-1495330553f8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.328140 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.327861    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/965caecb-d1eb-45a2-a48a-1495330553f8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.428793 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.428701    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/965caecb-d1eb-45a2-a48a-1495330553f8-config-out\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.428793 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.428746    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/965caecb-d1eb-45a2-a48a-1495330553f8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.428793 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.428766    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/965caecb-d1eb-45a2-a48a-1495330553f8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.428793 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.428784    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/965caecb-d1eb-45a2-a48a-1495330553f8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.429111 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.428904    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/965caecb-d1eb-45a2-a48a-1495330553f8-config\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.429111 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.428945    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/965caecb-d1eb-45a2-a48a-1495330553f8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.429111 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.428979    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/965caecb-d1eb-45a2-a48a-1495330553f8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.429111 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.429003    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/965caecb-d1eb-45a2-a48a-1495330553f8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.429111 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.429032    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/965caecb-d1eb-45a2-a48a-1495330553f8-web-config\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.429111 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.429081    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/965caecb-d1eb-45a2-a48a-1495330553f8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.429111 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.429103    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/965caecb-d1eb-45a2-a48a-1495330553f8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.432930 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.429139    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/965caecb-d1eb-45a2-a48a-1495330553f8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.432930 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.429162    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlqk8\" (UniqueName: \"kubernetes.io/projected/965caecb-d1eb-45a2-a48a-1495330553f8-kube-api-access-nlqk8\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.432930 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.429190    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/965caecb-d1eb-45a2-a48a-1495330553f8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.432930 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.429230    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/965caecb-d1eb-45a2-a48a-1495330553f8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.432930 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.429227    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/965caecb-d1eb-45a2-a48a-1495330553f8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.432930 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.429269    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/965caecb-d1eb-45a2-a48a-1495330553f8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.432930 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.429304    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/965caecb-d1eb-45a2-a48a-1495330553f8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.432930 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.429332    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/965caecb-d1eb-45a2-a48a-1495330553f8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.432930 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.429357    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/965caecb-d1eb-45a2-a48a-1495330553f8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.432930 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.429405    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/965caecb-d1eb-45a2-a48a-1495330553f8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.432930 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.429791    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/965caecb-d1eb-45a2-a48a-1495330553f8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.432930 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.429991    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/965caecb-d1eb-45a2-a48a-1495330553f8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.432930 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.430116    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/965caecb-d1eb-45a2-a48a-1495330553f8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.432930 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.431098    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/965caecb-d1eb-45a2-a48a-1495330553f8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.432930 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.431899    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/965caecb-d1eb-45a2-a48a-1495330553f8-config\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.432930 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.432628    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/965caecb-d1eb-45a2-a48a-1495330553f8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.432930 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.432718    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/965caecb-d1eb-45a2-a48a-1495330553f8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.433692 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.432722    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/965caecb-d1eb-45a2-a48a-1495330553f8-web-config\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.433692 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.432823    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/965caecb-d1eb-45a2-a48a-1495330553f8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.433692 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.433219    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/965caecb-d1eb-45a2-a48a-1495330553f8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.433692 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.433262    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/965caecb-d1eb-45a2-a48a-1495330553f8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.433692 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.433476    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/965caecb-d1eb-45a2-a48a-1495330553f8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.433934 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.433902    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/965caecb-d1eb-45a2-a48a-1495330553f8-config-out\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.434267 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.434240    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/965caecb-d1eb-45a2-a48a-1495330553f8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.434719 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.434691    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/965caecb-d1eb-45a2-a48a-1495330553f8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.434783 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.434718    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/965caecb-d1eb-45a2-a48a-1495330553f8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.435469 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.435453    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/965caecb-d1eb-45a2-a48a-1495330553f8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.435803 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.435785    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/965caecb-d1eb-45a2-a48a-1495330553f8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.440633 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.440613    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlqk8\" (UniqueName: \"kubernetes.io/projected/965caecb-d1eb-45a2-a48a-1495330553f8-kube-api-access-nlqk8\") pod \"prometheus-k8s-0\" (UID: \"965caecb-d1eb-45a2-a48a-1495330553f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.509471 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.509430    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:16.656320 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:16.656285    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 02 13:17:16.658253 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:17:16.658225    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod965caecb_d1eb_45a2_a48a_1495330553f8.slice/crio-eb67ef266b1d1441413f7a7da75ddb74a2bb270e1f5d2fb1c29591436e1549c6 WatchSource:0}: Error finding container eb67ef266b1d1441413f7a7da75ddb74a2bb270e1f5d2fb1c29591436e1549c6: Status 404 returned error can't find the container with id eb67ef266b1d1441413f7a7da75ddb74a2bb270e1f5d2fb1c29591436e1549c6
Jul 02 13:17:17.138788 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:17.138754    2583 generic.go:356] "Generic (PLEG): container finished" podID="965caecb-d1eb-45a2-a48a-1495330553f8" containerID="340e2348f33045925ca4a1d2199db5e738975a542ab3466793205e23310092dd" exitCode=0
Jul 02 13:17:17.139270 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:17.138842    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"965caecb-d1eb-45a2-a48a-1495330553f8","Type":"ContainerDied","Data":"340e2348f33045925ca4a1d2199db5e738975a542ab3466793205e23310092dd"}
Jul 02 13:17:17.139270 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:17.138929    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"965caecb-d1eb-45a2-a48a-1495330553f8","Type":"ContainerStarted","Data":"eb67ef266b1d1441413f7a7da75ddb74a2bb270e1f5d2fb1c29591436e1549c6"}
Jul 02 13:17:17.238093 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:17.238057    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5772ab56-d521-4af5-8c02-4d017569eb98" path="/var/lib/kubelet/pods/5772ab56-d521-4af5-8c02-4d017569eb98/volumes"
Jul 02 13:17:18.078004 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:18.077969    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:18.146039 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:18.145999    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"965caecb-d1eb-45a2-a48a-1495330553f8","Type":"ContainerStarted","Data":"c0655b77c28537c24ce6ec1a5e05e30f44d96a9499c0c1641f001bd734c0421e"}
Jul 02 13:17:18.146430 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:18.146107    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"965caecb-d1eb-45a2-a48a-1495330553f8","Type":"ContainerStarted","Data":"0d7d7993444f1c2aa92895b3f858ec8fab334333ba209a0896133edc356b20dc"}
Jul 02 13:17:18.146430 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:18.146134    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"965caecb-d1eb-45a2-a48a-1495330553f8","Type":"ContainerStarted","Data":"a962f50e23bb1e73a81c5da22baadb342b72a5ff4cd0e29a0f0883088f5987ad"}
Jul 02 13:17:18.146430 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:18.146163    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"965caecb-d1eb-45a2-a48a-1495330553f8","Type":"ContainerStarted","Data":"b72aee176e98f37e0b97a5daaccf7332679ece96493ce38ae67ec21d0b588abc"}
Jul 02 13:17:18.146430 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:18.146191    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"965caecb-d1eb-45a2-a48a-1495330553f8","Type":"ContainerStarted","Data":"73b8ef82ccbe8a74f6cfbcd553e51cca4e55c5cdaa63d3e9f47383dfed37a546"}
Jul 02 13:17:18.146430 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:18.146215    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"965caecb-d1eb-45a2-a48a-1495330553f8","Type":"ContainerStarted","Data":"c561e878893caaa10187cc3562de8e4a16c80c32044b0f21c19ef672fcffd472"}
Jul 02 13:17:18.157681 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:18.157657    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:18.180562 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:18.180512    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.165236686 podStartE2EDuration="2.180492384s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.255698ms" isStatefulPod=false podCreationTimestamp="2026-07-02 13:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:17:18.179674678 +0000 UTC m=+149.407585891" watchObservedRunningTime="2026-07-02 13:17:18.180492384 +0000 UTC m=+149.408403558"
Jul 02 13:17:21.510037 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:21.509981    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:17:22.694899 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:22.694853    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:22.798575 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:22.798509    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:22.901263 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:22.901220    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:23.767737 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:23.767696    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:23.870331 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:23.870283    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:23.925559 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:23.925521    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:31.928819 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:31.928783    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:38.232981 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:38.232942    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:17:38.334226 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:17:38.334195    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:18:16.510208 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:18:16.510163    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:18:16.526404 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:18:16.526367    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:18:17.420765 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:18:17.420734    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 02 13:18:31.711017 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:18:31.710976    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:18:31.784707 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:18:31.784663    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:18:32.000135 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:18:32.000041    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:18:32.111595 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:18:32.111564    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:18:32.174043 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:18:32.174004    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:18:56.340938 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:18:56.340875    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:18:56.441327 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:18:56.441289    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:18:58.821566 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:18:58.821526    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:18:58.952758 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:18:58.952714    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:19:04.467971 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:19:04.467935    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:19:04.736123 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:19:04.736038    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:19:05.117188 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:19:05.117158    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:19:05.723299 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:19:05.723262    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:19:05.963645 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:19:05.963600    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:19:06.103763 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:19:06.103688    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:19:06.202472 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:19:06.202439    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:19:06.510745 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:19:06.510714    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:19:06.711172 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:19:06.711122    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:19:06.920689 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:19:06.920657    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:19:07.019096 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:19:07.019058    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:19:19.254814 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:19:19.254739    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:19:19.328966 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:19:19.328933    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:19:31.560344 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:19:31.560312    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:19:34.049924 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:19:34.049892    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:19:49.176017 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:19:49.175985    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 13:19:49.179696 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:19:49.179669    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 13:19:49.187377 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:19:49.187353    2583 kubelet.go:1648] "Image garbage collection succeeded"
Jul 02 13:20:01.402617 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:20:01.402579    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:20:01.524776 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:20:01.524738    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:20:03.444261 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:20:03.444215    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:20:13.946645 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:20:13.946613    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:20:14.033779 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:20:14.033701    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:20:14.106549 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:20:14.106515    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:20:14.208017 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:20:14.207944    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:20:14.308652 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:20:14.308618    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:20:14.537636 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:20:14.537526    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:20:14.638806 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:20:14.638762    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:20:51.403120 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:20:51.403030    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:20:51.510283 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:20:51.510246    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:10.964349 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:10.964307    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:13.971949 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:13.971910    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:15.994566 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:15.994533    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:16.091455 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:16.091418    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:18.161850 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:18.161814    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:18.448681 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:18.448598    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:20.742642 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:20.742602    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:20.847144 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:20.847109    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:21.670770 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:21.670740    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:22.059882 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:22.059796    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:23.629211 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:23.629175    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:23.651922 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:23.651885    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5d99449ffd-b8llt"]
Jul 02 13:21:23.654232 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:23.654208    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5d99449ffd-b8llt"
Jul 02 13:21:23.656963 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:23.656940    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 02 13:21:23.657074 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:23.656989    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-79k77\""
Jul 02 13:21:23.658063 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:23.658041    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 02 13:21:23.673231 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:23.673193    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5d99449ffd-b8llt"]
Jul 02 13:21:23.768829 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:23.768788    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5719b470-94d0-4677-bb02-40d737e1701c-bound-sa-token\") pod \"cert-manager-cainjector-5d99449ffd-b8llt\" (UID: \"5719b470-94d0-4677-bb02-40d737e1701c\") " pod="cert-manager/cert-manager-cainjector-5d99449ffd-b8llt"
Jul 02 13:21:23.769001 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:23.768845    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r5fp\" (UniqueName: \"kubernetes.io/projected/5719b470-94d0-4677-bb02-40d737e1701c-kube-api-access-9r5fp\") pod \"cert-manager-cainjector-5d99449ffd-b8llt\" (UID: \"5719b470-94d0-4677-bb02-40d737e1701c\") " pod="cert-manager/cert-manager-cainjector-5d99449ffd-b8llt"
Jul 02 13:21:23.836300 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:23.836268    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:23.869413 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:23.869338    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5719b470-94d0-4677-bb02-40d737e1701c-bound-sa-token\") pod \"cert-manager-cainjector-5d99449ffd-b8llt\" (UID: \"5719b470-94d0-4677-bb02-40d737e1701c\") " pod="cert-manager/cert-manager-cainjector-5d99449ffd-b8llt"
Jul 02 13:21:23.869576 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:23.869446    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r5fp\" (UniqueName: \"kubernetes.io/projected/5719b470-94d0-4677-bb02-40d737e1701c-kube-api-access-9r5fp\") pod \"cert-manager-cainjector-5d99449ffd-b8llt\" (UID: \"5719b470-94d0-4677-bb02-40d737e1701c\") " pod="cert-manager/cert-manager-cainjector-5d99449ffd-b8llt"
Jul 02 13:21:23.888979 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:23.888906    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r5fp\" (UniqueName: \"kubernetes.io/projected/5719b470-94d0-4677-bb02-40d737e1701c-kube-api-access-9r5fp\") pod \"cert-manager-cainjector-5d99449ffd-b8llt\" (UID: \"5719b470-94d0-4677-bb02-40d737e1701c\") " pod="cert-manager/cert-manager-cainjector-5d99449ffd-b8llt"
Jul 02 13:21:23.888979 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:23.888917    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5719b470-94d0-4677-bb02-40d737e1701c-bound-sa-token\") pod \"cert-manager-cainjector-5d99449ffd-b8llt\" (UID: \"5719b470-94d0-4677-bb02-40d737e1701c\") " pod="cert-manager/cert-manager-cainjector-5d99449ffd-b8llt"
Jul 02 13:21:23.975102 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:23.975064    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5d99449ffd-b8llt"
Jul 02 13:21:23.986398 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:23.986350    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:24.106458 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:24.106353    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5d99449ffd-b8llt"]
Jul 02 13:21:24.109431 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:21:24.109406    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5719b470_94d0_4677_bb02_40d737e1701c.slice/crio-05457f56fee45eaeea7e78fe96c4c183aab8fb32d6d62db9c37691e811effa3b WatchSource:0}: Error finding container 05457f56fee45eaeea7e78fe96c4c183aab8fb32d6d62db9c37691e811effa3b: Status 404 returned error can't find the container with id 05457f56fee45eaeea7e78fe96c4c183aab8fb32d6d62db9c37691e811effa3b
Jul 02 13:21:24.111861 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:24.111845    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 13:21:24.190719 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:24.190635    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5d99449ffd-b8llt" event={"ID":"5719b470-94d0-4677-bb02-40d737e1701c","Type":"ContainerStarted","Data":"05457f56fee45eaeea7e78fe96c4c183aab8fb32d6d62db9c37691e811effa3b"}
Jul 02 13:21:26.250298 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:26.250262    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-7dc76cbb94-6x6wr"]
Jul 02 13:21:26.253271 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:26.253242    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-7dc76cbb94-6x6wr"
Jul 02 13:21:26.256435 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:26.256410    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-pb2xc\""
Jul 02 13:21:26.270599 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:26.270561    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-7dc76cbb94-6x6wr"]
Jul 02 13:21:26.390039 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:26.389971    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b50537c9-7f32-4af5-9b58-062e9b0ba7b7-bound-sa-token\") pod \"cert-manager-webhook-7dc76cbb94-6x6wr\" (UID: \"b50537c9-7f32-4af5-9b58-062e9b0ba7b7\") " pod="cert-manager/cert-manager-webhook-7dc76cbb94-6x6wr"
Jul 02 13:21:26.390213 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:26.390192    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftrfh\" (UniqueName: \"kubernetes.io/projected/b50537c9-7f32-4af5-9b58-062e9b0ba7b7-kube-api-access-ftrfh\") pod \"cert-manager-webhook-7dc76cbb94-6x6wr\" (UID: \"b50537c9-7f32-4af5-9b58-062e9b0ba7b7\") " pod="cert-manager/cert-manager-webhook-7dc76cbb94-6x6wr"
Jul 02 13:21:26.491240 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:26.491201    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftrfh\" (UniqueName: \"kubernetes.io/projected/b50537c9-7f32-4af5-9b58-062e9b0ba7b7-kube-api-access-ftrfh\") pod \"cert-manager-webhook-7dc76cbb94-6x6wr\" (UID: \"b50537c9-7f32-4af5-9b58-062e9b0ba7b7\") " pod="cert-manager/cert-manager-webhook-7dc76cbb94-6x6wr"
Jul 02 13:21:26.491468 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:26.491264    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b50537c9-7f32-4af5-9b58-062e9b0ba7b7-bound-sa-token\") pod \"cert-manager-webhook-7dc76cbb94-6x6wr\" (UID: \"b50537c9-7f32-4af5-9b58-062e9b0ba7b7\") " pod="cert-manager/cert-manager-webhook-7dc76cbb94-6x6wr"
Jul 02 13:21:26.504171 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:26.504071    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftrfh\" (UniqueName: \"kubernetes.io/projected/b50537c9-7f32-4af5-9b58-062e9b0ba7b7-kube-api-access-ftrfh\") pod \"cert-manager-webhook-7dc76cbb94-6x6wr\" (UID: \"b50537c9-7f32-4af5-9b58-062e9b0ba7b7\") " pod="cert-manager/cert-manager-webhook-7dc76cbb94-6x6wr"
Jul 02 13:21:26.504476 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:26.504443    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b50537c9-7f32-4af5-9b58-062e9b0ba7b7-bound-sa-token\") pod \"cert-manager-webhook-7dc76cbb94-6x6wr\" (UID: \"b50537c9-7f32-4af5-9b58-062e9b0ba7b7\") " pod="cert-manager/cert-manager-webhook-7dc76cbb94-6x6wr"
Jul 02 13:21:26.567103 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:26.567066    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-7dc76cbb94-6x6wr"
Jul 02 13:21:26.664078 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:26.664036    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:26.734554 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:26.734500    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-7dc76cbb94-6x6wr"]
Jul 02 13:21:28.213082 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:21:28.213041    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb50537c9_7f32_4af5_9b58_062e9b0ba7b7.slice/crio-5b729536d46e5410078dc115910cd89a00a29ce7206bd988e9a634c43056da69 WatchSource:0}: Error finding container 5b729536d46e5410078dc115910cd89a00a29ce7206bd988e9a634c43056da69: Status 404 returned error can't find the container with id 5b729536d46e5410078dc115910cd89a00a29ce7206bd988e9a634c43056da69
Jul 02 13:21:29.218775 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:29.218725    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5d99449ffd-b8llt" event={"ID":"5719b470-94d0-4677-bb02-40d737e1701c","Type":"ContainerStarted","Data":"9133aad2e49007b8ec97e40e7fc314b46a1c484881b71d4cfbe2696e0e0251c5"}
Jul 02 13:21:29.220071 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:29.220042    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-7dc76cbb94-6x6wr" event={"ID":"b50537c9-7f32-4af5-9b58-062e9b0ba7b7","Type":"ContainerStarted","Data":"5b729536d46e5410078dc115910cd89a00a29ce7206bd988e9a634c43056da69"}
Jul 02 13:21:29.237055 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:29.237001    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5d99449ffd-b8llt" podStartSLOduration=2.101677304 podStartE2EDuration="6.2369791s" totalImagesPullingTime="4.135301796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:21:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:21:29.235716003 +0000 UTC m=+400.463627178" watchObservedRunningTime="2026-07-02 13:21:29.2369791 +0000 UTC m=+400.464890275"
Jul 02 13:21:29.240004 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:29.239978    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:30.005350 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:30.005264    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:30.104722 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:30.104691    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:30.225734 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:30.225697    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-7dc76cbb94-6x6wr" event={"ID":"b50537c9-7f32-4af5-9b58-062e9b0ba7b7","Type":"ContainerStarted","Data":"fc8c58426f9dab48a49a3919c0263fcba2255b67d34b498b1526df207f53468f"}
Jul 02 13:21:30.226183 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:30.225789    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-7dc76cbb94-6x6wr"
Jul 02 13:21:30.251762 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:30.251713    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-7dc76cbb94-6x6wr" podStartSLOduration=2.7111778109999998 podStartE2EDuration="4.251696108s" totalImagesPullingTime="1.540518297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:21:30.249334834 +0000 UTC m=+401.477246008" watchObservedRunningTime="2026-07-02 13:21:30.251696108 +0000 UTC m=+401.479607282"
Jul 02 13:21:31.135987 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:31.135957    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-59c77684cc-smcs9"]
Jul 02 13:21:31.138407 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:31.138374    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-59c77684cc-smcs9"
Jul 02 13:21:31.141266 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:31.141247    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-mq7p2\""
Jul 02 13:21:31.150276 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:31.150248    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-59c77684cc-smcs9"]
Jul 02 13:21:31.232781 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:31.232749    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pzk4\" (UniqueName: \"kubernetes.io/projected/f7e09362-f89d-493f-bcf7-b7b70e6fe602-kube-api-access-9pzk4\") pod \"cert-manager-59c77684cc-smcs9\" (UID: \"f7e09362-f89d-493f-bcf7-b7b70e6fe602\") " pod="cert-manager/cert-manager-59c77684cc-smcs9"
Jul 02 13:21:31.233168 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:31.232867    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7e09362-f89d-493f-bcf7-b7b70e6fe602-bound-sa-token\") pod \"cert-manager-59c77684cc-smcs9\" (UID: \"f7e09362-f89d-493f-bcf7-b7b70e6fe602\") " pod="cert-manager/cert-manager-59c77684cc-smcs9"
Jul 02 13:21:31.333270 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:31.333237    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7e09362-f89d-493f-bcf7-b7b70e6fe602-bound-sa-token\") pod \"cert-manager-59c77684cc-smcs9\" (UID: \"f7e09362-f89d-493f-bcf7-b7b70e6fe602\") " pod="cert-manager/cert-manager-59c77684cc-smcs9"
Jul 02 13:21:31.333473 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:31.333288    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pzk4\" (UniqueName: \"kubernetes.io/projected/f7e09362-f89d-493f-bcf7-b7b70e6fe602-kube-api-access-9pzk4\") pod \"cert-manager-59c77684cc-smcs9\" (UID: \"f7e09362-f89d-493f-bcf7-b7b70e6fe602\") " pod="cert-manager/cert-manager-59c77684cc-smcs9"
Jul 02 13:21:31.341906 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:31.341881    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7e09362-f89d-493f-bcf7-b7b70e6fe602-bound-sa-token\") pod \"cert-manager-59c77684cc-smcs9\" (UID: \"f7e09362-f89d-493f-bcf7-b7b70e6fe602\") " pod="cert-manager/cert-manager-59c77684cc-smcs9"
Jul 02 13:21:31.342105 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:31.342086    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pzk4\" (UniqueName: \"kubernetes.io/projected/f7e09362-f89d-493f-bcf7-b7b70e6fe602-kube-api-access-9pzk4\") pod \"cert-manager-59c77684cc-smcs9\" (UID: \"f7e09362-f89d-493f-bcf7-b7b70e6fe602\") " pod="cert-manager/cert-manager-59c77684cc-smcs9"
Jul 02 13:21:31.447212 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:31.447090    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-59c77684cc-smcs9"
Jul 02 13:21:31.585226 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:31.585200    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-59c77684cc-smcs9"]
Jul 02 13:21:31.587312 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:21:31.587283    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7e09362_f89d_493f_bcf7_b7b70e6fe602.slice/crio-3a699c2ccab83002b835194211f347540b365d08b81a9a4e7291af51e95f49ad WatchSource:0}: Error finding container 3a699c2ccab83002b835194211f347540b365d08b81a9a4e7291af51e95f49ad: Status 404 returned error can't find the container with id 3a699c2ccab83002b835194211f347540b365d08b81a9a4e7291af51e95f49ad
Jul 02 13:21:32.235607 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:32.235569    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-59c77684cc-smcs9" event={"ID":"f7e09362-f89d-493f-bcf7-b7b70e6fe602","Type":"ContainerStarted","Data":"9935a5f8591db9f23f1bd20ecc16e2a40e47954531576b634b5495186c843b77"}
Jul 02 13:21:32.236022 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:32.235616    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-59c77684cc-smcs9" event={"ID":"f7e09362-f89d-493f-bcf7-b7b70e6fe602","Type":"ContainerStarted","Data":"3a699c2ccab83002b835194211f347540b365d08b81a9a4e7291af51e95f49ad"}
Jul 02 13:21:32.254660 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:32.254614    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-59c77684cc-smcs9" podStartSLOduration=1.2545978230000001 podStartE2EDuration="1.254597823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:21:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:21:32.253098708 +0000 UTC m=+403.481009882" watchObservedRunningTime="2026-07-02 13:21:32.254597823 +0000 UTC m=+403.482508997"
Jul 02 13:21:34.071299 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:34.071265    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:34.171474 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:34.171442    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:34.926922 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:34.926893    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:36.231730 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:36.231703    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-7dc76cbb94-6x6wr"
Jul 02 13:21:37.883804 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:37.883756    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:37.982825 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:37.982792    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:42.790632 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:42.790600    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:47.361689 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:47.361651    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:47.458039 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:47.458010    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:48.468667 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:48.468631    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:52.626572 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:52.626532    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:52.724542 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:52.724508    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:54.370448 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:54.370418    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:54.431416 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:54.431232    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:57.546994 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:57.546963    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:59.021367 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:59.021329    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:59.078283 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:59.078252    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-676677444b-x285c"]
Jul 02 13:21:59.084686 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:59.084665    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-676677444b-x285c"
Jul 02 13:21:59.087409 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:59.087364    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 02 13:21:59.087535 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:59.087422    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 02 13:21:59.087658 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:59.087643    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 02 13:21:59.088703 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:59.088685    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 02 13:21:59.088817 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:59.088707    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-5bgv5\""
Jul 02 13:21:59.088817 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:59.088785    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 02 13:21:59.095733 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:59.095709    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-676677444b-x285c"]
Jul 02 13:21:59.113356 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:59.113328    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:21:59.177793 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:59.177758    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21b94548-a1f6-42a4-be04-9de433ac4f8f-cert\") pod \"lws-controller-manager-676677444b-x285c\" (UID: \"21b94548-a1f6-42a4-be04-9de433ac4f8f\") " pod="openshift-lws-operator/lws-controller-manager-676677444b-x285c"
Jul 02 13:21:59.177793 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:59.177800    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/21b94548-a1f6-42a4-be04-9de433ac4f8f-manager-config\") pod \"lws-controller-manager-676677444b-x285c\" (UID: \"21b94548-a1f6-42a4-be04-9de433ac4f8f\") " pod="openshift-lws-operator/lws-controller-manager-676677444b-x285c"
Jul 02 13:21:59.178068 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:59.177827    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/21b94548-a1f6-42a4-be04-9de433ac4f8f-metrics-cert\") pod \"lws-controller-manager-676677444b-x285c\" (UID: \"21b94548-a1f6-42a4-be04-9de433ac4f8f\") " pod="openshift-lws-operator/lws-controller-manager-676677444b-x285c"
Jul 02 13:21:59.178068 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:59.177894    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp55m\" (UniqueName: \"kubernetes.io/projected/21b94548-a1f6-42a4-be04-9de433ac4f8f-kube-api-access-cp55m\") pod \"lws-controller-manager-676677444b-x285c\" (UID: \"21b94548-a1f6-42a4-be04-9de433ac4f8f\") " pod="openshift-lws-operator/lws-controller-manager-676677444b-x285c"
Jul 02 13:21:59.279483 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:59.279372    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21b94548-a1f6-42a4-be04-9de433ac4f8f-cert\") pod \"lws-controller-manager-676677444b-x285c\" (UID: \"21b94548-a1f6-42a4-be04-9de433ac4f8f\") " pod="openshift-lws-operator/lws-controller-manager-676677444b-x285c"
Jul 02 13:21:59.279483 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:59.279446    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/21b94548-a1f6-42a4-be04-9de433ac4f8f-manager-config\") pod \"lws-controller-manager-676677444b-x285c\" (UID: \"21b94548-a1f6-42a4-be04-9de433ac4f8f\") " pod="openshift-lws-operator/lws-controller-manager-676677444b-x285c"
Jul 02 13:21:59.279483 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:59.279473    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/21b94548-a1f6-42a4-be04-9de433ac4f8f-metrics-cert\") pod \"lws-controller-manager-676677444b-x285c\" (UID: \"21b94548-a1f6-42a4-be04-9de433ac4f8f\") " pod="openshift-lws-operator/lws-controller-manager-676677444b-x285c"
Jul 02 13:21:59.279689 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:59.279498    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp55m\" (UniqueName: \"kubernetes.io/projected/21b94548-a1f6-42a4-be04-9de433ac4f8f-kube-api-access-cp55m\") pod \"lws-controller-manager-676677444b-x285c\" (UID: \"21b94548-a1f6-42a4-be04-9de433ac4f8f\") " pod="openshift-lws-operator/lws-controller-manager-676677444b-x285c"
Jul 02 13:21:59.280337 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:59.280307    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/21b94548-a1f6-42a4-be04-9de433ac4f8f-manager-config\") pod \"lws-controller-manager-676677444b-x285c\" (UID: \"21b94548-a1f6-42a4-be04-9de433ac4f8f\") " pod="openshift-lws-operator/lws-controller-manager-676677444b-x285c"
Jul 02 13:21:59.281864 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:59.281844    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/21b94548-a1f6-42a4-be04-9de433ac4f8f-cert\") pod \"lws-controller-manager-676677444b-x285c\" (UID: \"21b94548-a1f6-42a4-be04-9de433ac4f8f\") " pod="openshift-lws-operator/lws-controller-manager-676677444b-x285c"
Jul 02 13:21:59.281939 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:59.281893    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/21b94548-a1f6-42a4-be04-9de433ac4f8f-metrics-cert\") pod \"lws-controller-manager-676677444b-x285c\" (UID: \"21b94548-a1f6-42a4-be04-9de433ac4f8f\") " pod="openshift-lws-operator/lws-controller-manager-676677444b-x285c"
Jul 02 13:21:59.306143 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:59.306110    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp55m\" (UniqueName: \"kubernetes.io/projected/21b94548-a1f6-42a4-be04-9de433ac4f8f-kube-api-access-cp55m\") pod \"lws-controller-manager-676677444b-x285c\" (UID: \"21b94548-a1f6-42a4-be04-9de433ac4f8f\") " pod="openshift-lws-operator/lws-controller-manager-676677444b-x285c"
Jul 02 13:21:59.394134 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:59.394099    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-676677444b-x285c"
Jul 02 13:21:59.535191 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:59.535163    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-676677444b-x285c"]
Jul 02 13:21:59.537842 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:21:59.537816    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b94548_a1f6_42a4_be04_9de433ac4f8f.slice/crio-179ea59bcb255a39a4ace99ca66ffdc1baa005776712ab703bae2a93351bf501 WatchSource:0}: Error finding container 179ea59bcb255a39a4ace99ca66ffdc1baa005776712ab703bae2a93351bf501: Status 404 returned error can't find the container with id 179ea59bcb255a39a4ace99ca66ffdc1baa005776712ab703bae2a93351bf501
Jul 02 13:21:59.989534 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:21:59.989506    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:00.093444 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:00.093417    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:00.354399 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:00.354306    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-676677444b-x285c" event={"ID":"21b94548-a1f6-42a4-be04-9de433ac4f8f","Type":"ContainerStarted","Data":"179ea59bcb255a39a4ace99ca66ffdc1baa005776712ab703bae2a93351bf501"}
Jul 02 13:22:03.371725 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:03.371692    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-676677444b-x285c" event={"ID":"21b94548-a1f6-42a4-be04-9de433ac4f8f","Type":"ContainerStarted","Data":"55e53db384e757ca34f865c46f3da6cd9dd9a81e485232148034e595c2a0b78b"}
Jul 02 13:22:03.372117 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:03.371833    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-676677444b-x285c"
Jul 02 13:22:03.394992 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:03.394863    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-676677444b-x285c" podStartSLOduration=0.853389419 podStartE2EDuration="4.39484446s" totalImagesPullingTime="3.541455041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:21:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:22:03.39355685 +0000 UTC m=+434.621468027" watchObservedRunningTime="2026-07-02 13:22:03.39484446 +0000 UTC m=+434.622755634"
Jul 02 13:22:10.405563 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:10.405519    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:12.804198 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:12.804163    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:13.213063 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:13.213019    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:14.377783 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:14.377753    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-676677444b-x285c"
Jul 02 13:22:14.930328 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:14.930292    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6d85f6564-h7pt5"]
Jul 02 13:22:14.934164 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:14.934143    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6d85f6564-h7pt5"
Jul 02 13:22:14.937084 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:14.937064    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 02 13:22:14.937211 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:14.937104    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 02 13:22:14.937211 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:14.937146    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-jhknc\""
Jul 02 13:22:14.945135 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:14.945112    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6d85f6564-h7pt5"]
Jul 02 13:22:15.043269 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:15.043236    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:15.128089 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:15.128043    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnsqf\" (UniqueName: \"kubernetes.io/projected/0b801638-1cfe-4e8f-a194-d5f2a8c714e2-kube-api-access-mnsqf\") pod \"authorino-operator-6d85f6564-h7pt5\" (UID: \"0b801638-1cfe-4e8f-a194-d5f2a8c714e2\") " pod="kuadrant-system/authorino-operator-6d85f6564-h7pt5"
Jul 02 13:22:15.149608 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:15.149550    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:15.228574 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:15.228482    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnsqf\" (UniqueName: \"kubernetes.io/projected/0b801638-1cfe-4e8f-a194-d5f2a8c714e2-kube-api-access-mnsqf\") pod \"authorino-operator-6d85f6564-h7pt5\" (UID: \"0b801638-1cfe-4e8f-a194-d5f2a8c714e2\") " pod="kuadrant-system/authorino-operator-6d85f6564-h7pt5"
Jul 02 13:22:15.237769 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:15.237739    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnsqf\" (UniqueName: \"kubernetes.io/projected/0b801638-1cfe-4e8f-a194-d5f2a8c714e2-kube-api-access-mnsqf\") pod \"authorino-operator-6d85f6564-h7pt5\" (UID: \"0b801638-1cfe-4e8f-a194-d5f2a8c714e2\") " pod="kuadrant-system/authorino-operator-6d85f6564-h7pt5"
Jul 02 13:22:15.244521 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:15.244500    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6d85f6564-h7pt5"
Jul 02 13:22:15.396896 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:15.396869    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6d85f6564-h7pt5"]
Jul 02 13:22:15.398952 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:22:15.398916    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b801638_1cfe_4e8f_a194_d5f2a8c714e2.slice/crio-0092b3def5f7ba1aa83d3f600b0c322624d908a12d835beefc6d87f3fc61bd32 WatchSource:0}: Error finding container 0092b3def5f7ba1aa83d3f600b0c322624d908a12d835beefc6d87f3fc61bd32: Status 404 returned error can't find the container with id 0092b3def5f7ba1aa83d3f600b0c322624d908a12d835beefc6d87f3fc61bd32
Jul 02 13:22:15.431833 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:15.431796    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6d85f6564-h7pt5" event={"ID":"0b801638-1cfe-4e8f-a194-d5f2a8c714e2","Type":"ContainerStarted","Data":"0092b3def5f7ba1aa83d3f600b0c322624d908a12d835beefc6d87f3fc61bd32"}
Jul 02 13:22:15.797754 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:15.797723    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:16.509954 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:16.509916    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:17.633587 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:17.633536    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:17.820697 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:17.820662    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:18.451428 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:18.451327    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6d85f6564-h7pt5" event={"ID":"0b801638-1cfe-4e8f-a194-d5f2a8c714e2","Type":"ContainerStarted","Data":"95607539daa2e9d6f167e00899b07e8d1161e4286f3c316bb8180fd67d9c8444"}
Jul 02 13:22:18.451582 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:18.451455    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6d85f6564-h7pt5"
Jul 02 13:22:18.484145 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:18.484089    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6d85f6564-h7pt5" podStartSLOduration=1.7470513410000001 podStartE2EDuration="4.484073585s" totalImagesPullingTime="2.737022244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:22:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:22:18.483485871 +0000 UTC m=+449.711397046" watchObservedRunningTime="2026-07-02 13:22:18.484073585 +0000 UTC m=+449.711984758"
Jul 02 13:22:18.678698 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:18.678663    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:18.780324 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:18.780247    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:18.879836 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:18.879799    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:24.107515 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:24.107479    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:24.205471 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:24.205439    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:27.618930 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:27.618894    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:27.729683 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:27.729649    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:29.459662 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:29.459632    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6d85f6564-h7pt5"
Jul 02 13:22:29.887873 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:29.887838    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:30.083594 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:30.083562    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:30.183292 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:30.183214    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:36.527731 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:36.527694    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:36.636888 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:36.636858    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:36.731921 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:36.731888    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:43.188175 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:43.188136    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:43.292344 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:43.292297    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:48.595266 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:48.595228    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:48.695685 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:48.695653    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:48.992971 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:48.992937    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:49.137207 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:49.137176    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:49.206156 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:49.206125    2583 scope.go:117] "RemoveContainer" containerID="b411e9a05b50c7209cbf1d2c1f8f438ec6e4457401b00ae96a90f2687c2d91d7"
Jul 02 13:22:49.214250 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:49.214224    2583 scope.go:117] "RemoveContainer" containerID="e344ae5150a86bb2342e9cd4ef14f6c9902cfbd4fe85012ce50b9b4bdb7ae7f0"
Jul 02 13:22:49.221658 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:49.221641    2583 scope.go:117] "RemoveContainer" containerID="76c8f8c577ab675b334167508900629fa74ac333a5e4e3a003d80938b77bc813"
Jul 02 13:22:49.229415 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:49.229378    2583 scope.go:117] "RemoveContainer" containerID="c7d3dadeb32b528bae0c69d223f0ed263240d57a73e6df86a17e331733bddf60"
Jul 02 13:22:49.237483 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:49.237451    2583 scope.go:117] "RemoveContainer" containerID="1c586f4447e899f63eb93699c7377ca8579b1ab5e2f9466f4babbfb0bb1275b4"
Jul 02 13:22:49.244924 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:49.244902    2583 scope.go:117] "RemoveContainer" containerID="69dbdde8164c3e3c5eb3946a35e9f6dca70b2785162bb3498bcb7f4fb78a7082"
Jul 02 13:22:49.252040 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:49.252021    2583 scope.go:117] "RemoveContainer" containerID="580d6f6bb79535caffffba94bf5b4ddeca1c77f84377f47815d14cd12f362eee"
Jul 02 13:22:50.580408 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:50.580353    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:50.680928 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:50.680896    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:52.822993 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:52.822959    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:52.929351 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:52.929316    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:57.537276 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:57.537239    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:57.635946 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:57.635912    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:22:57.697568 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:22:57.697532    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:23:12.848161 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:12.848124    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:23:12.855140 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:12.855110    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-hhnwz"]
Jul 02 13:23:12.857607 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:12.857591    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-hhnwz"
Jul 02 13:23:12.861359 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:12.861335    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 02 13:23:12.861585 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:12.861572    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-tzvvp\""
Jul 02 13:23:12.870521 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:12.870497    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-hhnwz"]
Jul 02 13:23:12.940602 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:12.940570    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d53b52c1-96cc-431d-b5d6-92d022602c8e-config-file\") pod \"limitador-limitador-ff68ff5cb-hhnwz\" (UID: \"d53b52c1-96cc-431d-b5d6-92d022602c8e\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-hhnwz"
Jul 02 13:23:12.940780 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:12.940608    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48sk7\" (UniqueName: \"kubernetes.io/projected/d53b52c1-96cc-431d-b5d6-92d022602c8e-kube-api-access-48sk7\") pod \"limitador-limitador-ff68ff5cb-hhnwz\" (UID: \"d53b52c1-96cc-431d-b5d6-92d022602c8e\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-hhnwz"
Jul 02 13:23:12.956297 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:12.956257    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-hhnwz"]
Jul 02 13:23:12.989581 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:12.989546    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:23:13.041641 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:13.041600    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d53b52c1-96cc-431d-b5d6-92d022602c8e-config-file\") pod \"limitador-limitador-ff68ff5cb-hhnwz\" (UID: \"d53b52c1-96cc-431d-b5d6-92d022602c8e\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-hhnwz"
Jul 02 13:23:13.041641 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:13.041641    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48sk7\" (UniqueName: \"kubernetes.io/projected/d53b52c1-96cc-431d-b5d6-92d022602c8e-kube-api-access-48sk7\") pod \"limitador-limitador-ff68ff5cb-hhnwz\" (UID: \"d53b52c1-96cc-431d-b5d6-92d022602c8e\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-hhnwz"
Jul 02 13:23:13.042286 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:13.042267    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d53b52c1-96cc-431d-b5d6-92d022602c8e-config-file\") pod \"limitador-limitador-ff68ff5cb-hhnwz\" (UID: \"d53b52c1-96cc-431d-b5d6-92d022602c8e\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-hhnwz"
Jul 02 13:23:13.053504 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:13.053468    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48sk7\" (UniqueName: \"kubernetes.io/projected/d53b52c1-96cc-431d-b5d6-92d022602c8e-kube-api-access-48sk7\") pod \"limitador-limitador-ff68ff5cb-hhnwz\" (UID: \"d53b52c1-96cc-431d-b5d6-92d022602c8e\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-hhnwz"
Jul 02 13:23:13.124714 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:13.124676    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:23:13.167740 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:13.167706    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-hhnwz"
Jul 02 13:23:13.311333 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:13.311301    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-hhnwz"]
Jul 02 13:23:13.314366 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:23:13.314338    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd53b52c1_96cc_431d_b5d6_92d022602c8e.slice/crio-98a12e27fb1fbd88a862c73221cd7d6026b6012250a9e22d390fc7759d5e3f55 WatchSource:0}: Error finding container 98a12e27fb1fbd88a862c73221cd7d6026b6012250a9e22d390fc7759d5e3f55: Status 404 returned error can't find the container with id 98a12e27fb1fbd88a862c73221cd7d6026b6012250a9e22d390fc7759d5e3f55
Jul 02 13:23:13.706457 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:13.706422    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-hhnwz" event={"ID":"d53b52c1-96cc-431d-b5d6-92d022602c8e","Type":"ContainerStarted","Data":"98a12e27fb1fbd88a862c73221cd7d6026b6012250a9e22d390fc7759d5e3f55"}
Jul 02 13:23:13.724761 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:13.724717    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b6c89bb85-zgfqr"]
Jul 02 13:23:13.727863 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:13.727841    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-zgfqr"
Jul 02 13:23:13.731315 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:13.731287    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rflsm\""
Jul 02 13:23:13.736354 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:13.736331    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-zgfqr"]
Jul 02 13:23:13.748420 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:13.748370    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mb6w\" (UniqueName: \"kubernetes.io/projected/f5e5cbb4-39a9-4ad3-8fb0-a9264d7d797c-kube-api-access-7mb6w\") pod \"authorino-b6c89bb85-zgfqr\" (UID: \"f5e5cbb4-39a9-4ad3-8fb0-a9264d7d797c\") " pod="kuadrant-system/authorino-b6c89bb85-zgfqr"
Jul 02 13:23:13.772096 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:13.772058    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:23:13.849727 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:13.849689    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mb6w\" (UniqueName: \"kubernetes.io/projected/f5e5cbb4-39a9-4ad3-8fb0-a9264d7d797c-kube-api-access-7mb6w\") pod \"authorino-b6c89bb85-zgfqr\" (UID: \"f5e5cbb4-39a9-4ad3-8fb0-a9264d7d797c\") " pod="kuadrant-system/authorino-b6c89bb85-zgfqr"
Jul 02 13:23:13.850542 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:13.850520    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8df697c9f-4ccrk"]
Jul 02 13:23:13.853008 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:13.852984    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-4ccrk"
Jul 02 13:23:13.862996 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:13.862970    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mb6w\" (UniqueName: \"kubernetes.io/projected/f5e5cbb4-39a9-4ad3-8fb0-a9264d7d797c-kube-api-access-7mb6w\") pod \"authorino-b6c89bb85-zgfqr\" (UID: \"f5e5cbb4-39a9-4ad3-8fb0-a9264d7d797c\") " pod="kuadrant-system/authorino-b6c89bb85-zgfqr"
Jul 02 13:23:13.865491 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:13.865453    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8df697c9f-4ccrk"]
Jul 02 13:23:13.869191 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:13.869167    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:23:13.950474 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:13.950438    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prpps\" (UniqueName: \"kubernetes.io/projected/c0b9d077-877e-473f-bc44-ae195856b4fe-kube-api-access-prpps\") pod \"authorino-8df697c9f-4ccrk\" (UID: \"c0b9d077-877e-473f-bc44-ae195856b4fe\") " pod="kuadrant-system/authorino-8df697c9f-4ccrk"
Jul 02 13:23:14.038579 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:14.038487    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-zgfqr"
Jul 02 13:23:14.052583 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:14.052025    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prpps\" (UniqueName: \"kubernetes.io/projected/c0b9d077-877e-473f-bc44-ae195856b4fe-kube-api-access-prpps\") pod \"authorino-8df697c9f-4ccrk\" (UID: \"c0b9d077-877e-473f-bc44-ae195856b4fe\") " pod="kuadrant-system/authorino-8df697c9f-4ccrk"
Jul 02 13:23:14.062890 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:14.062854    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prpps\" (UniqueName: \"kubernetes.io/projected/c0b9d077-877e-473f-bc44-ae195856b4fe-kube-api-access-prpps\") pod \"authorino-8df697c9f-4ccrk\" (UID: \"c0b9d077-877e-473f-bc44-ae195856b4fe\") " pod="kuadrant-system/authorino-8df697c9f-4ccrk"
Jul 02 13:23:14.161493 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:14.161461    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-4ccrk"
Jul 02 13:23:14.205952 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:14.205747    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-zgfqr"]
Jul 02 13:23:14.350713 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:14.350643    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8df697c9f-4ccrk"]
Jul 02 13:23:14.353172 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:23:14.353148    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0b9d077_877e_473f_bc44_ae195856b4fe.slice/crio-4ce62a4cba418037f7e8918b5a365a5a24965800d5004af693c285d5e5b1bff8 WatchSource:0}: Error finding container 4ce62a4cba418037f7e8918b5a365a5a24965800d5004af693c285d5e5b1bff8: Status 404 returned error can't find the container with id 4ce62a4cba418037f7e8918b5a365a5a24965800d5004af693c285d5e5b1bff8
Jul 02 13:23:14.713452 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:14.713417    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-4ccrk" event={"ID":"c0b9d077-877e-473f-bc44-ae195856b4fe","Type":"ContainerStarted","Data":"4ce62a4cba418037f7e8918b5a365a5a24965800d5004af693c285d5e5b1bff8"}
Jul 02 13:23:14.714744 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:14.714718    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-zgfqr" event={"ID":"f5e5cbb4-39a9-4ad3-8fb0-a9264d7d797c","Type":"ContainerStarted","Data":"b40150d7684133537dc16cce57605daef3d8f980a510160560ae03299c39ad76"}
Jul 02 13:23:18.741048 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:18.740927    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-zgfqr" event={"ID":"f5e5cbb4-39a9-4ad3-8fb0-a9264d7d797c","Type":"ContainerStarted","Data":"2aa5aa53f5784b98ec57b7c07e928c6e6351e715086488fd89e10f019aba0ac6"}
Jul 02 13:23:18.742517 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:18.742491    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-hhnwz" event={"ID":"d53b52c1-96cc-431d-b5d6-92d022602c8e","Type":"ContainerStarted","Data":"4b571000bd1776c36e8ca35c5b7d7ff153cdf7087074c503c9e04accd5404773"}
Jul 02 13:23:18.742649 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:18.742563    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-ff68ff5cb-hhnwz"
Jul 02 13:23:18.743935 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:18.743915    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-4ccrk" event={"ID":"c0b9d077-877e-473f-bc44-ae195856b4fe","Type":"ContainerStarted","Data":"b0677e560e742442744789418de9ecfdae1aac04b12e38b9e2c111301d9e7921"}
Jul 02 13:23:18.755929 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:18.755882    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b6c89bb85-zgfqr" podStartSLOduration=1.559074285 podStartE2EDuration="5.755866656s" totalImagesPullingTime="4.196792371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:23:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:23:18.75564984 +0000 UTC m=+509.983561015" watchObservedRunningTime="2026-07-02 13:23:18.755866656 +0000 UTC m=+509.983777807"
Jul 02 13:23:18.773925 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:18.773865    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8df697c9f-4ccrk" podStartSLOduration=1.590545262 podStartE2EDuration="5.773848284s" totalImagesPullingTime="4.183303022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:23:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:23:18.772299392 +0000 UTC m=+510.000210567" watchObservedRunningTime="2026-07-02 13:23:18.773848284 +0000 UTC m=+510.001759481"
Jul 02 13:23:18.795195 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:18.795140    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-ff68ff5cb-hhnwz" podStartSLOduration=1.7165699220000001 podStartE2EDuration="6.795123717s" totalImagesPullingTime="5.078553795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:23:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:23:18.793962028 +0000 UTC m=+510.021873203" watchObservedRunningTime="2026-07-02 13:23:18.795123717 +0000 UTC m=+510.023034891"
Jul 02 13:23:18.802091 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:18.802058    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-zgfqr"]
Jul 02 13:23:20.754266 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:20.754225    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-b6c89bb85-zgfqr" podUID="f5e5cbb4-39a9-4ad3-8fb0-a9264d7d797c" containerName="authorino" containerID="cri-o://2aa5aa53f5784b98ec57b7c07e928c6e6351e715086488fd89e10f019aba0ac6" gracePeriod=30
Jul 02 13:23:21.010135 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:21.010073    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-zgfqr"
Jul 02 13:23:21.114833 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:21.114792    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mb6w\" (UniqueName: \"kubernetes.io/projected/f5e5cbb4-39a9-4ad3-8fb0-a9264d7d797c-kube-api-access-7mb6w\") pod \"f5e5cbb4-39a9-4ad3-8fb0-a9264d7d797c\" (UID: \"f5e5cbb4-39a9-4ad3-8fb0-a9264d7d797c\") "
Jul 02 13:23:21.116942 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:21.116910    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5e5cbb4-39a9-4ad3-8fb0-a9264d7d797c-kube-api-access-7mb6w" (OuterVolumeSpecName: "kube-api-access-7mb6w") pod "f5e5cbb4-39a9-4ad3-8fb0-a9264d7d797c" (UID: "f5e5cbb4-39a9-4ad3-8fb0-a9264d7d797c"). InnerVolumeSpecName "kube-api-access-7mb6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:23:21.215572 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:21.215531    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mb6w\" (UniqueName: \"kubernetes.io/projected/f5e5cbb4-39a9-4ad3-8fb0-a9264d7d797c-kube-api-access-7mb6w\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:23:21.760804 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:21.760768    2583 generic.go:356] "Generic (PLEG): container finished" podID="f5e5cbb4-39a9-4ad3-8fb0-a9264d7d797c" containerID="2aa5aa53f5784b98ec57b7c07e928c6e6351e715086488fd89e10f019aba0ac6" exitCode=0
Jul 02 13:23:21.761222 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:21.760826    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-zgfqr"
Jul 02 13:23:21.761222 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:21.760858    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-zgfqr" event={"ID":"f5e5cbb4-39a9-4ad3-8fb0-a9264d7d797c","Type":"ContainerDied","Data":"2aa5aa53f5784b98ec57b7c07e928c6e6351e715086488fd89e10f019aba0ac6"}
Jul 02 13:23:21.761222 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:21.760907    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-zgfqr" event={"ID":"f5e5cbb4-39a9-4ad3-8fb0-a9264d7d797c","Type":"ContainerDied","Data":"b40150d7684133537dc16cce57605daef3d8f980a510160560ae03299c39ad76"}
Jul 02 13:23:21.761222 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:21.760924    2583 scope.go:117] "RemoveContainer" containerID="2aa5aa53f5784b98ec57b7c07e928c6e6351e715086488fd89e10f019aba0ac6"
Jul 02 13:23:21.769326 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:21.769308    2583 scope.go:117] "RemoveContainer" containerID="2aa5aa53f5784b98ec57b7c07e928c6e6351e715086488fd89e10f019aba0ac6"
Jul 02 13:23:21.769697 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:23:21.769671    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aa5aa53f5784b98ec57b7c07e928c6e6351e715086488fd89e10f019aba0ac6\": container with ID starting with 2aa5aa53f5784b98ec57b7c07e928c6e6351e715086488fd89e10f019aba0ac6 not found: ID does not exist" containerID="2aa5aa53f5784b98ec57b7c07e928c6e6351e715086488fd89e10f019aba0ac6"
Jul 02 13:23:21.769752 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:21.769706    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aa5aa53f5784b98ec57b7c07e928c6e6351e715086488fd89e10f019aba0ac6"} err="failed to get container status \"2aa5aa53f5784b98ec57b7c07e928c6e6351e715086488fd89e10f019aba0ac6\": rpc error: code = NotFound desc = could not find container \"2aa5aa53f5784b98ec57b7c07e928c6e6351e715086488fd89e10f019aba0ac6\": container with ID starting with 2aa5aa53f5784b98ec57b7c07e928c6e6351e715086488fd89e10f019aba0ac6 not found: ID does not exist"
Jul 02 13:23:21.780027 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:21.780000    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-zgfqr"]
Jul 02 13:23:21.784344 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:21.784320    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-zgfqr"]
Jul 02 13:23:23.234103 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:23.234065    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5e5cbb4-39a9-4ad3-8fb0-a9264d7d797c" path="/var/lib/kubelet/pods/f5e5cbb4-39a9-4ad3-8fb0-a9264d7d797c/volumes"
Jul 02 13:23:29.750639 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:29.750607    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-ff68ff5cb-hhnwz"
Jul 02 13:23:29.809994 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:29.809961    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-hhnwz"]
Jul 02 13:23:29.810174 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:29.810132    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-ff68ff5cb-hhnwz" podUID="d53b52c1-96cc-431d-b5d6-92d022602c8e" containerName="limitador" containerID="cri-o://4b571000bd1776c36e8ca35c5b7d7ff153cdf7087074c503c9e04accd5404773" gracePeriod=30
Jul 02 13:23:30.361215 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:30.361187    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-hhnwz"
Jul 02 13:23:30.502809 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:30.502716    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48sk7\" (UniqueName: \"kubernetes.io/projected/d53b52c1-96cc-431d-b5d6-92d022602c8e-kube-api-access-48sk7\") pod \"d53b52c1-96cc-431d-b5d6-92d022602c8e\" (UID: \"d53b52c1-96cc-431d-b5d6-92d022602c8e\") "
Jul 02 13:23:30.502809 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:30.502767    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d53b52c1-96cc-431d-b5d6-92d022602c8e-config-file\") pod \"d53b52c1-96cc-431d-b5d6-92d022602c8e\" (UID: \"d53b52c1-96cc-431d-b5d6-92d022602c8e\") "
Jul 02 13:23:30.503194 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:30.503163    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d53b52c1-96cc-431d-b5d6-92d022602c8e-config-file" (OuterVolumeSpecName: "config-file") pod "d53b52c1-96cc-431d-b5d6-92d022602c8e" (UID: "d53b52c1-96cc-431d-b5d6-92d022602c8e"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:23:30.504879 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:30.504855    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d53b52c1-96cc-431d-b5d6-92d022602c8e-kube-api-access-48sk7" (OuterVolumeSpecName: "kube-api-access-48sk7") pod "d53b52c1-96cc-431d-b5d6-92d022602c8e" (UID: "d53b52c1-96cc-431d-b5d6-92d022602c8e"). InnerVolumeSpecName "kube-api-access-48sk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:23:30.603688 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:30.603646    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48sk7\" (UniqueName: \"kubernetes.io/projected/d53b52c1-96cc-431d-b5d6-92d022602c8e-kube-api-access-48sk7\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:23:30.603688 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:30.603679    2583 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d53b52c1-96cc-431d-b5d6-92d022602c8e-config-file\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:23:30.808077 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:30.807994    2583 generic.go:356] "Generic (PLEG): container finished" podID="d53b52c1-96cc-431d-b5d6-92d022602c8e" containerID="4b571000bd1776c36e8ca35c5b7d7ff153cdf7087074c503c9e04accd5404773" exitCode=0
Jul 02 13:23:30.808077 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:30.808040    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-hhnwz"
Jul 02 13:23:30.808626 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:30.808080    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-hhnwz" event={"ID":"d53b52c1-96cc-431d-b5d6-92d022602c8e","Type":"ContainerDied","Data":"4b571000bd1776c36e8ca35c5b7d7ff153cdf7087074c503c9e04accd5404773"}
Jul 02 13:23:30.808626 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:30.808137    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-hhnwz" event={"ID":"d53b52c1-96cc-431d-b5d6-92d022602c8e","Type":"ContainerDied","Data":"98a12e27fb1fbd88a862c73221cd7d6026b6012250a9e22d390fc7759d5e3f55"}
Jul 02 13:23:30.808626 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:30.808166    2583 scope.go:117] "RemoveContainer" containerID="4b571000bd1776c36e8ca35c5b7d7ff153cdf7087074c503c9e04accd5404773"
Jul 02 13:23:30.816666 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:30.816499    2583 scope.go:117] "RemoveContainer" containerID="4b571000bd1776c36e8ca35c5b7d7ff153cdf7087074c503c9e04accd5404773"
Jul 02 13:23:30.816808 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:23:30.816790    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b571000bd1776c36e8ca35c5b7d7ff153cdf7087074c503c9e04accd5404773\": container with ID starting with 4b571000bd1776c36e8ca35c5b7d7ff153cdf7087074c503c9e04accd5404773 not found: ID does not exist" containerID="4b571000bd1776c36e8ca35c5b7d7ff153cdf7087074c503c9e04accd5404773"
Jul 02 13:23:30.816850 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:30.816818    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b571000bd1776c36e8ca35c5b7d7ff153cdf7087074c503c9e04accd5404773"} err="failed to get container status \"4b571000bd1776c36e8ca35c5b7d7ff153cdf7087074c503c9e04accd5404773\": rpc error: code = NotFound desc = could not find container \"4b571000bd1776c36e8ca35c5b7d7ff153cdf7087074c503c9e04accd5404773\": container with ID starting with 4b571000bd1776c36e8ca35c5b7d7ff153cdf7087074c503c9e04accd5404773 not found: ID does not exist"
Jul 02 13:23:30.831514 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:30.831483    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-hhnwz"]
Jul 02 13:23:30.835132 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:30.835108    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-hhnwz"]
Jul 02 13:23:31.233702 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:31.233666    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d53b52c1-96cc-431d-b5d6-92d022602c8e" path="/var/lib/kubelet/pods/d53b52c1-96cc-431d-b5d6-92d022602c8e/volumes"
Jul 02 13:23:35.251383 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:35.251348    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:23:35.796809 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:35.796771    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:23:35.895240 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:35.895200    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:23:40.004597 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:40.004554    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c77957575-zf5kz"]
Jul 02 13:23:40.005103 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:40.004988    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d53b52c1-96cc-431d-b5d6-92d022602c8e" containerName="limitador"
Jul 02 13:23:40.005103 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:40.005008    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d53b52c1-96cc-431d-b5d6-92d022602c8e" containerName="limitador"
Jul 02 13:23:40.005103 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:40.005052    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5e5cbb4-39a9-4ad3-8fb0-a9264d7d797c" containerName="authorino"
Jul 02 13:23:40.005103 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:40.005060    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5e5cbb4-39a9-4ad3-8fb0-a9264d7d797c" containerName="authorino"
Jul 02 13:23:40.005271 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:40.005144    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d53b52c1-96cc-431d-b5d6-92d022602c8e" containerName="limitador"
Jul 02 13:23:40.005271 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:40.005163    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5e5cbb4-39a9-4ad3-8fb0-a9264d7d797c" containerName="authorino"
Jul 02 13:23:40.009424 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:40.009405    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c77957575-zf5kz"
Jul 02 13:23:40.012612 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:40.012592    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 02 13:23:40.019087 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:40.019062    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c77957575-zf5kz"]
Jul 02 13:23:40.089457 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:40.089419    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5857b5b0-7614-43f3-bc37-f6f6d88d0880-tls-cert\") pod \"authorino-7c77957575-zf5kz\" (UID: \"5857b5b0-7614-43f3-bc37-f6f6d88d0880\") " pod="kuadrant-system/authorino-7c77957575-zf5kz"
Jul 02 13:23:40.089627 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:40.089465    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j25c2\" (UniqueName: \"kubernetes.io/projected/5857b5b0-7614-43f3-bc37-f6f6d88d0880-kube-api-access-j25c2\") pod \"authorino-7c77957575-zf5kz\" (UID: \"5857b5b0-7614-43f3-bc37-f6f6d88d0880\") " pod="kuadrant-system/authorino-7c77957575-zf5kz"
Jul 02 13:23:40.190745 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:40.190708    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5857b5b0-7614-43f3-bc37-f6f6d88d0880-tls-cert\") pod \"authorino-7c77957575-zf5kz\" (UID: \"5857b5b0-7614-43f3-bc37-f6f6d88d0880\") " pod="kuadrant-system/authorino-7c77957575-zf5kz"
Jul 02 13:23:40.190925 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:40.190752    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j25c2\" (UniqueName: \"kubernetes.io/projected/5857b5b0-7614-43f3-bc37-f6f6d88d0880-kube-api-access-j25c2\") pod \"authorino-7c77957575-zf5kz\" (UID: \"5857b5b0-7614-43f3-bc37-f6f6d88d0880\") " pod="kuadrant-system/authorino-7c77957575-zf5kz"
Jul 02 13:23:40.193211 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:40.193177    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5857b5b0-7614-43f3-bc37-f6f6d88d0880-tls-cert\") pod \"authorino-7c77957575-zf5kz\" (UID: \"5857b5b0-7614-43f3-bc37-f6f6d88d0880\") " pod="kuadrant-system/authorino-7c77957575-zf5kz"
Jul 02 13:23:40.199968 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:40.199943    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j25c2\" (UniqueName: \"kubernetes.io/projected/5857b5b0-7614-43f3-bc37-f6f6d88d0880-kube-api-access-j25c2\") pod \"authorino-7c77957575-zf5kz\" (UID: \"5857b5b0-7614-43f3-bc37-f6f6d88d0880\") " pod="kuadrant-system/authorino-7c77957575-zf5kz"
Jul 02 13:23:40.319351 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:40.319259    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c77957575-zf5kz"
Jul 02 13:23:40.461141 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:40.461114    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c77957575-zf5kz"]
Jul 02 13:23:40.463406 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:23:40.463357    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5857b5b0_7614_43f3_bc37_f6f6d88d0880.slice/crio-c28397847de16d5673176a3a04029ae635ef24def56d63545b5f2544042d6fed WatchSource:0}: Error finding container c28397847de16d5673176a3a04029ae635ef24def56d63545b5f2544042d6fed: Status 404 returned error can't find the container with id c28397847de16d5673176a3a04029ae635ef24def56d63545b5f2544042d6fed
Jul 02 13:23:40.856945 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:40.856866    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c77957575-zf5kz" event={"ID":"5857b5b0-7614-43f3-bc37-f6f6d88d0880","Type":"ContainerStarted","Data":"c28397847de16d5673176a3a04029ae635ef24def56d63545b5f2544042d6fed"}
Jul 02 13:23:41.862886 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:41.862848    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c77957575-zf5kz" event={"ID":"5857b5b0-7614-43f3-bc37-f6f6d88d0880","Type":"ContainerStarted","Data":"e87bfbd69e7a651ab60ec9f901d1e13b7941a7e2ad48512d2599a6ea869ab639"}
Jul 02 13:23:41.882232 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:41.882183    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7c77957575-zf5kz" podStartSLOduration=2.421225117 podStartE2EDuration="2.882166241s" totalImagesPullingTime="460.941124ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:23:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:23:41.880661102 +0000 UTC m=+533.108572277" watchObservedRunningTime="2026-07-02 13:23:41.882166241 +0000 UTC m=+533.110077415"
Jul 02 13:23:41.908373 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:41.908331    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8df697c9f-4ccrk"]
Jul 02 13:23:41.908591 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:41.908564    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-8df697c9f-4ccrk" podUID="c0b9d077-877e-473f-bc44-ae195856b4fe" containerName="authorino" containerID="cri-o://b0677e560e742442744789418de9ecfdae1aac04b12e38b9e2c111301d9e7921" gracePeriod=30
Jul 02 13:23:42.170471 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:42.170444    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-4ccrk"
Jul 02 13:23:42.308517 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:42.308485    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prpps\" (UniqueName: \"kubernetes.io/projected/c0b9d077-877e-473f-bc44-ae195856b4fe-kube-api-access-prpps\") pod \"c0b9d077-877e-473f-bc44-ae195856b4fe\" (UID: \"c0b9d077-877e-473f-bc44-ae195856b4fe\") "
Jul 02 13:23:42.310563 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:42.310537    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b9d077-877e-473f-bc44-ae195856b4fe-kube-api-access-prpps" (OuterVolumeSpecName: "kube-api-access-prpps") pod "c0b9d077-877e-473f-bc44-ae195856b4fe" (UID: "c0b9d077-877e-473f-bc44-ae195856b4fe"). InnerVolumeSpecName "kube-api-access-prpps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:23:42.409832 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:42.409742    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prpps\" (UniqueName: \"kubernetes.io/projected/c0b9d077-877e-473f-bc44-ae195856b4fe-kube-api-access-prpps\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:23:42.868467 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:42.868433    2583 generic.go:356] "Generic (PLEG): container finished" podID="c0b9d077-877e-473f-bc44-ae195856b4fe" containerID="b0677e560e742442744789418de9ecfdae1aac04b12e38b9e2c111301d9e7921" exitCode=0
Jul 02 13:23:42.868941 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:42.868475    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-4ccrk"
Jul 02 13:23:42.868941 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:42.868514    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-4ccrk" event={"ID":"c0b9d077-877e-473f-bc44-ae195856b4fe","Type":"ContainerDied","Data":"b0677e560e742442744789418de9ecfdae1aac04b12e38b9e2c111301d9e7921"}
Jul 02 13:23:42.868941 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:42.868555    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-4ccrk" event={"ID":"c0b9d077-877e-473f-bc44-ae195856b4fe","Type":"ContainerDied","Data":"4ce62a4cba418037f7e8918b5a365a5a24965800d5004af693c285d5e5b1bff8"}
Jul 02 13:23:42.868941 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:42.868572    2583 scope.go:117] "RemoveContainer" containerID="b0677e560e742442744789418de9ecfdae1aac04b12e38b9e2c111301d9e7921"
Jul 02 13:23:42.877263 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:42.877243    2583 scope.go:117] "RemoveContainer" containerID="b0677e560e742442744789418de9ecfdae1aac04b12e38b9e2c111301d9e7921"
Jul 02 13:23:42.877540 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:23:42.877513    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0677e560e742442744789418de9ecfdae1aac04b12e38b9e2c111301d9e7921\": container with ID starting with b0677e560e742442744789418de9ecfdae1aac04b12e38b9e2c111301d9e7921 not found: ID does not exist" containerID="b0677e560e742442744789418de9ecfdae1aac04b12e38b9e2c111301d9e7921"
Jul 02 13:23:42.877641 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:42.877548    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0677e560e742442744789418de9ecfdae1aac04b12e38b9e2c111301d9e7921"} err="failed to get container status \"b0677e560e742442744789418de9ecfdae1aac04b12e38b9e2c111301d9e7921\": rpc error: code = NotFound desc = could not find container \"b0677e560e742442744789418de9ecfdae1aac04b12e38b9e2c111301d9e7921\": container with ID starting with b0677e560e742442744789418de9ecfdae1aac04b12e38b9e2c111301d9e7921 not found: ID does not exist"
Jul 02 13:23:42.890674 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:42.890647    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8df697c9f-4ccrk"]
Jul 02 13:23:42.894688 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:42.894665    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8df697c9f-4ccrk"]
Jul 02 13:23:43.235047 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:43.234971    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0b9d077-877e-473f-bc44-ae195856b4fe" path="/var/lib/kubelet/pods/c0b9d077-877e-473f-bc44-ae195856b4fe/volumes"
Jul 02 13:23:49.055642 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:23:49.055605    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:24:00.468700 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:00.468662    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:24:19.309456 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:19.309414    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:24:19.352740 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:19.352708    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-jl6r2"]
Jul 02 13:24:19.353071 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:19.353058    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0b9d077-877e-473f-bc44-ae195856b4fe" containerName="authorino"
Jul 02 13:24:19.353120 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:19.353072    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0b9d077-877e-473f-bc44-ae195856b4fe" containerName="authorino"
Jul 02 13:24:19.353159 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:19.353146    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0b9d077-877e-473f-bc44-ae195856b4fe" containerName="authorino"
Jul 02 13:24:19.355100 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:19.355084    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-jl6r2"
Jul 02 13:24:19.357926 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:19.357896    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-2cqzf\""
Jul 02 13:24:19.358070 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:19.357904    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 02 13:24:19.358976 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:19.358960    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 02 13:24:19.359064 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:19.358974    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 02 13:24:19.368698 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:19.368672    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-jl6r2"]
Jul 02 13:24:19.390683 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:19.390638    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:24:19.449597 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:19.449563    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d680a83-8ceb-4eb0-b7fc-311c1e1ccc94-tls-certs\") pod \"model-serving-api-fd65d7d6-jl6r2\" (UID: \"4d680a83-8ceb-4eb0-b7fc-311c1e1ccc94\") " pod="kserve/model-serving-api-fd65d7d6-jl6r2"
Jul 02 13:24:19.449808 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:19.449619    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knk75\" (UniqueName: \"kubernetes.io/projected/4d680a83-8ceb-4eb0-b7fc-311c1e1ccc94-kube-api-access-knk75\") pod \"model-serving-api-fd65d7d6-jl6r2\" (UID: \"4d680a83-8ceb-4eb0-b7fc-311c1e1ccc94\") " pod="kserve/model-serving-api-fd65d7d6-jl6r2"
Jul 02 13:24:19.550838 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:19.550796    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d680a83-8ceb-4eb0-b7fc-311c1e1ccc94-tls-certs\") pod \"model-serving-api-fd65d7d6-jl6r2\" (UID: \"4d680a83-8ceb-4eb0-b7fc-311c1e1ccc94\") " pod="kserve/model-serving-api-fd65d7d6-jl6r2"
Jul 02 13:24:19.551010 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:19.550863    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knk75\" (UniqueName: \"kubernetes.io/projected/4d680a83-8ceb-4eb0-b7fc-311c1e1ccc94-kube-api-access-knk75\") pod \"model-serving-api-fd65d7d6-jl6r2\" (UID: \"4d680a83-8ceb-4eb0-b7fc-311c1e1ccc94\") " pod="kserve/model-serving-api-fd65d7d6-jl6r2"
Jul 02 13:24:19.551010 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:24:19.550940    2583 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 02 13:24:19.551138 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:24:19.551016    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d680a83-8ceb-4eb0-b7fc-311c1e1ccc94-tls-certs podName:4d680a83-8ceb-4eb0-b7fc-311c1e1ccc94 nodeName:}" failed. No retries permitted until 2026-07-02 13:24:20.050998664 +0000 UTC m=+571.278909815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4d680a83-8ceb-4eb0-b7fc-311c1e1ccc94-tls-certs") pod "model-serving-api-fd65d7d6-jl6r2" (UID: "4d680a83-8ceb-4eb0-b7fc-311c1e1ccc94") : secret "model-serving-api-tls" not found
Jul 02 13:24:19.559839 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:19.559772    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knk75\" (UniqueName: \"kubernetes.io/projected/4d680a83-8ceb-4eb0-b7fc-311c1e1ccc94-kube-api-access-knk75\") pod \"model-serving-api-fd65d7d6-jl6r2\" (UID: \"4d680a83-8ceb-4eb0-b7fc-311c1e1ccc94\") " pod="kserve/model-serving-api-fd65d7d6-jl6r2"
Jul 02 13:24:19.736644 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:19.736615    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:24:19.837077 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:19.836984    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:24:20.043366 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:20.043314    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:24:20.057745 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:20.057716    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d680a83-8ceb-4eb0-b7fc-311c1e1ccc94-tls-certs\") pod \"model-serving-api-fd65d7d6-jl6r2\" (UID: \"4d680a83-8ceb-4eb0-b7fc-311c1e1ccc94\") " pod="kserve/model-serving-api-fd65d7d6-jl6r2"
Jul 02 13:24:20.060089 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:20.060069    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d680a83-8ceb-4eb0-b7fc-311c1e1ccc94-tls-certs\") pod \"model-serving-api-fd65d7d6-jl6r2\" (UID: \"4d680a83-8ceb-4eb0-b7fc-311c1e1ccc94\") " pod="kserve/model-serving-api-fd65d7d6-jl6r2"
Jul 02 13:24:20.128411 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:20.128362    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:24:20.265341 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:20.265306    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-jl6r2"
Jul 02 13:24:20.423320 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:20.423249    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-jl6r2"]
Jul 02 13:24:20.426602 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:24:20.426573    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d680a83_8ceb_4eb0_b7fc_311c1e1ccc94.slice/crio-b8b2f98c5b62dbbbfcda1936738a0a88a05995755941a513d0e4228e8deff54f WatchSource:0}: Error finding container b8b2f98c5b62dbbbfcda1936738a0a88a05995755941a513d0e4228e8deff54f: Status 404 returned error can't find the container with id b8b2f98c5b62dbbbfcda1936738a0a88a05995755941a513d0e4228e8deff54f
Jul 02 13:24:21.055908 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:21.055870    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-jl6r2" event={"ID":"4d680a83-8ceb-4eb0-b7fc-311c1e1ccc94","Type":"ContainerStarted","Data":"b8b2f98c5b62dbbbfcda1936738a0a88a05995755941a513d0e4228e8deff54f"}
Jul 02 13:24:23.067820 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:23.067722    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-jl6r2" event={"ID":"4d680a83-8ceb-4eb0-b7fc-311c1e1ccc94","Type":"ContainerStarted","Data":"ef90c7d6e95199878cc9901aef5183a783531413036b6d1c59209358b8de933e"}
Jul 02 13:24:23.067820 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:23.067793    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-jl6r2"
Jul 02 13:24:23.091020 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:23.090961    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-jl6r2" podStartSLOduration=1.722523286 podStartE2EDuration="4.09094055s" totalImagesPullingTime="2.368417264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:24:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:24:23.089984386 +0000 UTC m=+574.317895562" watchObservedRunningTime="2026-07-02 13:24:23.09094055 +0000 UTC m=+574.318851728"
Jul 02 13:24:28.968696 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:28.968664    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:24:34.077328 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:34.077293    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-jl6r2"
Jul 02 13:24:36.403904 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:36.403865    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-qw5vv"]
Jul 02 13:24:36.407370 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:36.407348    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qw5vv"
Jul 02 13:24:36.409933 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:36.409911    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 02 13:24:36.410021 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:36.409964    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-pp7s2\""
Jul 02 13:24:36.417293 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:36.417266    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-qw5vv"]
Jul 02 13:24:36.504941 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:36.504903    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0-model-cache\") pod \"s3-init-qw5vv\" (UID: \"2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0\") " pod="kserve/s3-init-qw5vv"
Jul 02 13:24:36.504941 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:36.504943    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px69w\" (UniqueName: \"kubernetes.io/projected/2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0-kube-api-access-px69w\") pod \"s3-init-qw5vv\" (UID: \"2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0\") " pod="kserve/s3-init-qw5vv"
Jul 02 13:24:36.605639 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:36.605600    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0-model-cache\") pod \"s3-init-qw5vv\" (UID: \"2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0\") " pod="kserve/s3-init-qw5vv"
Jul 02 13:24:36.605639 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:36.605643    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px69w\" (UniqueName: \"kubernetes.io/projected/2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0-kube-api-access-px69w\") pod \"s3-init-qw5vv\" (UID: \"2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0\") " pod="kserve/s3-init-qw5vv"
Jul 02 13:24:36.606016 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:36.605996    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0-model-cache\") pod \"s3-init-qw5vv\" (UID: \"2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0\") " pod="kserve/s3-init-qw5vv"
Jul 02 13:24:36.614729 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:36.614696    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px69w\" (UniqueName: \"kubernetes.io/projected/2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0-kube-api-access-px69w\") pod \"s3-init-qw5vv\" (UID: \"2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0\") " pod="kserve/s3-init-qw5vv"
Jul 02 13:24:36.716562 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:36.716473    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qw5vv"
Jul 02 13:24:36.856876 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:36.856839    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-qw5vv"]
Jul 02 13:24:36.859251 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:24:36.859220    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cfaa043_cd0e_416e_a6bb_84b0c6aa4ad0.slice/crio-c7797cc68cb9f25373cd1bb1e5cdaced81950fa7781f1928997fc3b5f8c88e1c WatchSource:0}: Error finding container c7797cc68cb9f25373cd1bb1e5cdaced81950fa7781f1928997fc3b5f8c88e1c: Status 404 returned error can't find the container with id c7797cc68cb9f25373cd1bb1e5cdaced81950fa7781f1928997fc3b5f8c88e1c
Jul 02 13:24:37.137246 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:37.137208    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qw5vv" event={"ID":"2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0","Type":"ContainerStarted","Data":"c7797cc68cb9f25373cd1bb1e5cdaced81950fa7781f1928997fc3b5f8c88e1c"}
Jul 02 13:24:41.159615 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:41.159578    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qw5vv" event={"ID":"2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0","Type":"ContainerStarted","Data":"dac14ba3d691e223a0acd5ce5c7f7f6fa8f0faab29fa6ac7f158006737e96a89"}
Jul 02 13:24:43.171663 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:43.171627    2583 generic.go:356] "Generic (PLEG): container finished" podID="2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0" containerID="dac14ba3d691e223a0acd5ce5c7f7f6fa8f0faab29fa6ac7f158006737e96a89" exitCode=0
Jul 02 13:24:43.172037 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:43.171705    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qw5vv" event={"ID":"2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0","Type":"ContainerDied","Data":"dac14ba3d691e223a0acd5ce5c7f7f6fa8f0faab29fa6ac7f158006737e96a89"}
Jul 02 13:24:49.204781 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:49.204739    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qw5vv" event={"ID":"2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0","Type":"ContainerStarted","Data":"4c306834beab3ee24b5a4952d6fba7b3015388eaf9db35affb08a2dc09f46710"}
Jul 02 13:24:49.216692 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:49.216660    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 13:24:49.217568 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:49.217540    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 13:24:49.223849 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:24:49.223795    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-qw5vv" podStartSLOduration=2.545363364 podStartE2EDuration="13.22377846s" totalImagesPullingTime="9.191279328s" totalInitContainerRuntime="1.487135768s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:24:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:24:49.223318757 +0000 UTC m=+600.451229945" watchObservedRunningTime="2026-07-02 13:24:49.22377846 +0000 UTC m=+600.451689636"
Jul 02 13:25:54.516198 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:25:54.516161    2583 generic.go:356] "Generic (PLEG): container finished" podID="2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0" containerID="4c306834beab3ee24b5a4952d6fba7b3015388eaf9db35affb08a2dc09f46710" exitCode=0
Jul 02 13:25:54.516664 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:25:54.516230    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qw5vv" event={"ID":"2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0","Type":"ContainerDied","Data":"4c306834beab3ee24b5a4952d6fba7b3015388eaf9db35affb08a2dc09f46710"}
Jul 02 13:25:55.651794 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:25:55.651769    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qw5vv"
Jul 02 13:25:55.751584 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:25:55.751545    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-px69w\" (UniqueName: \"kubernetes.io/projected/2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0-kube-api-access-px69w\") pod \"2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0\" (UID: \"2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0\") "
Jul 02 13:25:55.751775 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:25:55.751611    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0-model-cache\") pod \"2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0\" (UID: \"2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0\") "
Jul 02 13:25:55.753950 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:25:55.753921    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0-kube-api-access-px69w" (OuterVolumeSpecName: "kube-api-access-px69w") pod "2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0" (UID: "2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0"). InnerVolumeSpecName "kube-api-access-px69w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:25:55.771475 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:25:55.771442    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0-model-cache" (OuterVolumeSpecName: "model-cache") pod "2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0" (UID: "2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:25:55.852721 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:25:55.852629    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px69w\" (UniqueName: \"kubernetes.io/projected/2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0-kube-api-access-px69w\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:25:55.852721 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:25:55.852662    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:25:56.528337 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:25:56.528312    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-qw5vv"
Jul 02 13:25:56.528337 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:25:56.528323    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-qw5vv" event={"ID":"2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0","Type":"ContainerDied","Data":"c7797cc68cb9f25373cd1bb1e5cdaced81950fa7781f1928997fc3b5f8c88e1c"}
Jul 02 13:25:56.528611 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:25:56.528356    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7797cc68cb9f25373cd1bb1e5cdaced81950fa7781f1928997fc3b5f8c88e1c"
Jul 02 13:25:56.930785 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:25:56.930746    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:25:58.165593 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:25:58.165556    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:26:06.477304 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:06.477269    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:26:09.080630 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.080595    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"]
Jul 02 13:26:09.081178 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.081157    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0" containerName="s3-init"
Jul 02 13:26:09.081260 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.081181    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0" containerName="s3-init"
Jul 02 13:26:09.081260 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.081228    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0" containerName="download-hf-model"
Jul 02 13:26:09.081260 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.081239    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0" containerName="download-hf-model"
Jul 02 13:26:09.081433 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.081326    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0" containerName="s3-init"
Jul 02 13:26:09.084671 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.084649    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:09.088485 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.088462    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 13:26:09.088746 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.088724    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 13:26:09.088812 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.088742    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-2d6pw\""
Jul 02 13:26:09.088812 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.088796    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 02 13:26:09.103787 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.103759    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"]
Jul 02 13:26:09.122128 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.122094    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:26:09.173751 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.173713    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91bed416-a9bf-4dcf-af67-bceba2ece5ff-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gg6n8\" (UID: \"91bed416-a9bf-4dcf-af67-bceba2ece5ff\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:09.173751 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.173758    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91bed416-a9bf-4dcf-af67-bceba2ece5ff-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gg6n8\" (UID: \"91bed416-a9bf-4dcf-af67-bceba2ece5ff\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:09.174010 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.173822    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91bed416-a9bf-4dcf-af67-bceba2ece5ff-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gg6n8\" (UID: \"91bed416-a9bf-4dcf-af67-bceba2ece5ff\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:09.174010 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.173859    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91bed416-a9bf-4dcf-af67-bceba2ece5ff-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gg6n8\" (UID: \"91bed416-a9bf-4dcf-af67-bceba2ece5ff\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:09.174010 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.173896    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91bed416-a9bf-4dcf-af67-bceba2ece5ff-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gg6n8\" (UID: \"91bed416-a9bf-4dcf-af67-bceba2ece5ff\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:09.174010 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.173954    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91bed416-a9bf-4dcf-af67-bceba2ece5ff-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gg6n8\" (UID: \"91bed416-a9bf-4dcf-af67-bceba2ece5ff\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:09.174010 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.173977    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91bed416-a9bf-4dcf-af67-bceba2ece5ff-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gg6n8\" (UID: \"91bed416-a9bf-4dcf-af67-bceba2ece5ff\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:09.174010 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.174003    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdq7m\" (UniqueName: \"kubernetes.io/projected/91bed416-a9bf-4dcf-af67-bceba2ece5ff-kube-api-access-kdq7m\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gg6n8\" (UID: \"91bed416-a9bf-4dcf-af67-bceba2ece5ff\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:09.174225 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.174097    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91bed416-a9bf-4dcf-af67-bceba2ece5ff-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gg6n8\" (UID: \"91bed416-a9bf-4dcf-af67-bceba2ece5ff\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:09.217763 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.217727    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:26:09.275554 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.275512    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91bed416-a9bf-4dcf-af67-bceba2ece5ff-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gg6n8\" (UID: \"91bed416-a9bf-4dcf-af67-bceba2ece5ff\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:09.275554 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.275551    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91bed416-a9bf-4dcf-af67-bceba2ece5ff-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gg6n8\" (UID: \"91bed416-a9bf-4dcf-af67-bceba2ece5ff\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:09.275805 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.275571    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdq7m\" (UniqueName: \"kubernetes.io/projected/91bed416-a9bf-4dcf-af67-bceba2ece5ff-kube-api-access-kdq7m\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gg6n8\" (UID: \"91bed416-a9bf-4dcf-af67-bceba2ece5ff\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:09.275805 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.275600    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91bed416-a9bf-4dcf-af67-bceba2ece5ff-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gg6n8\" (UID: \"91bed416-a9bf-4dcf-af67-bceba2ece5ff\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:09.275805 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.275635    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91bed416-a9bf-4dcf-af67-bceba2ece5ff-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gg6n8\" (UID: \"91bed416-a9bf-4dcf-af67-bceba2ece5ff\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:09.275805 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.275654    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91bed416-a9bf-4dcf-af67-bceba2ece5ff-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gg6n8\" (UID: \"91bed416-a9bf-4dcf-af67-bceba2ece5ff\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:09.276025 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.275810    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91bed416-a9bf-4dcf-af67-bceba2ece5ff-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gg6n8\" (UID: \"91bed416-a9bf-4dcf-af67-bceba2ece5ff\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:09.276025 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.275845    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91bed416-a9bf-4dcf-af67-bceba2ece5ff-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gg6n8\" (UID: \"91bed416-a9bf-4dcf-af67-bceba2ece5ff\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:09.276025 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.275874    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91bed416-a9bf-4dcf-af67-bceba2ece5ff-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gg6n8\" (UID: \"91bed416-a9bf-4dcf-af67-bceba2ece5ff\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:09.276188 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.276062    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91bed416-a9bf-4dcf-af67-bceba2ece5ff-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gg6n8\" (UID: \"91bed416-a9bf-4dcf-af67-bceba2ece5ff\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:09.276354 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.276325    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91bed416-a9bf-4dcf-af67-bceba2ece5ff-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gg6n8\" (UID: \"91bed416-a9bf-4dcf-af67-bceba2ece5ff\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:09.276354 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.276342    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91bed416-a9bf-4dcf-af67-bceba2ece5ff-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gg6n8\" (UID: \"91bed416-a9bf-4dcf-af67-bceba2ece5ff\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:09.276542 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.276413    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91bed416-a9bf-4dcf-af67-bceba2ece5ff-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gg6n8\" (UID: \"91bed416-a9bf-4dcf-af67-bceba2ece5ff\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:09.276542 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.276426    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91bed416-a9bf-4dcf-af67-bceba2ece5ff-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gg6n8\" (UID: \"91bed416-a9bf-4dcf-af67-bceba2ece5ff\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:09.277873 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.277855    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91bed416-a9bf-4dcf-af67-bceba2ece5ff-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gg6n8\" (UID: \"91bed416-a9bf-4dcf-af67-bceba2ece5ff\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:09.278106 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.278089    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91bed416-a9bf-4dcf-af67-bceba2ece5ff-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gg6n8\" (UID: \"91bed416-a9bf-4dcf-af67-bceba2ece5ff\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:09.287179 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.287129    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91bed416-a9bf-4dcf-af67-bceba2ece5ff-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gg6n8\" (UID: \"91bed416-a9bf-4dcf-af67-bceba2ece5ff\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:09.287291 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.287186    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdq7m\" (UniqueName: \"kubernetes.io/projected/91bed416-a9bf-4dcf-af67-bceba2ece5ff-kube-api-access-kdq7m\") pod \"router-gateway-1-openshift-default-54c47cdfc7-gg6n8\" (UID: \"91bed416-a9bf-4dcf-af67-bceba2ece5ff\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:09.395573 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.395532    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:09.551129 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.551092    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"]
Jul 02 13:26:09.554995 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:26:09.554962    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91bed416_a9bf_4dcf_af67_bceba2ece5ff.slice/crio-b7983d1baf2c05942a6b2974b0a1babdb82302c64ee0fdb435952ac3e761f69a WatchSource:0}: Error finding container b7983d1baf2c05942a6b2974b0a1babdb82302c64ee0fdb435952ac3e761f69a: Status 404 returned error can't find the container with id b7983d1baf2c05942a6b2974b0a1babdb82302c64ee0fdb435952ac3e761f69a
Jul 02 13:26:09.591209 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:09.591171    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8" event={"ID":"91bed416-a9bf-4dcf-af67-bceba2ece5ff","Type":"ContainerStarted","Data":"b7983d1baf2c05942a6b2974b0a1babdb82302c64ee0fdb435952ac3e761f69a"}
Jul 02 13:26:12.577422 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:12.577367    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 02 13:26:12.577721 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:12.577460    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 02 13:26:12.577721 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:12.577507    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 02 13:26:13.065552 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:13.065513    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:26:13.612554 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:13.612515    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8" event={"ID":"91bed416-a9bf-4dcf-af67-bceba2ece5ff","Type":"ContainerStarted","Data":"485e4081edbeef9aa08b6856667c5249ed3cb01ba661ac5c5063a5dce566698a"}
Jul 02 13:26:13.638272 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:13.638219    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8" podStartSLOduration=1.617737112 podStartE2EDuration="4.638203525s" totalImagesPullingTime="3.020466413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:26:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:26:13.636481979 +0000 UTC m=+684.864393149" watchObservedRunningTime="2026-07-02 13:26:13.638203525 +0000 UTC m=+684.866114698"
Jul 02 13:26:14.395967 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:14.395929    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:14.400723 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:14.400698    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:14.618208 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:14.618174    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:14.619174 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:14.619156    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:15.679652 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:15.679622    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:26:15.747806 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:15.747760    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:26:31.926248 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:31.926196    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:26:35.646070 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:35.646015    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:26:35.720945 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:35.720906    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:26:38.368992 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:38.368954    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:26:41.762850 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:41.762812    2583 generic.go:356] "Generic (PLEG): container finished" podID="91bed416-a9bf-4dcf-af67-bceba2ece5ff" containerID="485e4081edbeef9aa08b6856667c5249ed3cb01ba661ac5c5063a5dce566698a" exitCode=0
Jul 02 13:26:41.763414 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:41.762874    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8" event={"ID":"91bed416-a9bf-4dcf-af67-bceba2ece5ff","Type":"ContainerDied","Data":"485e4081edbeef9aa08b6856667c5249ed3cb01ba661ac5c5063a5dce566698a"}
Jul 02 13:26:41.763414 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:41.763372    2583 scope.go:117] "RemoveContainer" containerID="485e4081edbeef9aa08b6856667c5249ed3cb01ba661ac5c5063a5dce566698a"
Jul 02 13:26:41.764453 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:41.764418    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 02 13:26:41.764522 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:41.764493    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 02 13:26:41.764559 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:41.764526    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 02 13:26:42.428307 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.428268    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"]
Jul 02 13:26:42.431130 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.431102    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"
Jul 02 13:26:42.436752 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.436723    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:26:42.438432 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.438406    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7s8sh\""
Jul 02 13:26:42.446218 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.446196    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jul 02 13:26:42.481904 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.481866    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-tmp-dir\") pod \"precise-prefix-cache-test-kserve-5767855db4-jhrhr\" (UID: \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"
Jul 02 13:26:42.482086 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.481908    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5gsw\" (UniqueName: \"kubernetes.io/projected/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-kube-api-access-w5gsw\") pod \"precise-prefix-cache-test-kserve-5767855db4-jhrhr\" (UID: \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"
Jul 02 13:26:42.482086 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.482042    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-tls-certs\") pod \"precise-prefix-cache-test-kserve-5767855db4-jhrhr\" (UID: \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"
Jul 02 13:26:42.482086 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.482074    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-home\") pod \"precise-prefix-cache-test-kserve-5767855db4-jhrhr\" (UID: \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"
Jul 02 13:26:42.482227 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.482104    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-model-cache\") pod \"precise-prefix-cache-test-kserve-5767855db4-jhrhr\" (UID: \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"
Jul 02 13:26:42.482227 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.482121    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-dshm\") pod \"precise-prefix-cache-test-kserve-5767855db4-jhrhr\" (UID: \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"
Jul 02 13:26:42.482227 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.482143    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-5767855db4-jhrhr\" (UID: \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"
Jul 02 13:26:42.497771 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.497735    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"]
Jul 02 13:26:42.536999 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.536965    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:26:42.584074 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.584026    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-tls-certs\") pod \"precise-prefix-cache-test-kserve-5767855db4-jhrhr\" (UID: \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"
Jul 02 13:26:42.584250 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.584096    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-home\") pod \"precise-prefix-cache-test-kserve-5767855db4-jhrhr\" (UID: \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"
Jul 02 13:26:42.584250 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.584140    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-model-cache\") pod \"precise-prefix-cache-test-kserve-5767855db4-jhrhr\" (UID: \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"
Jul 02 13:26:42.584250 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.584171    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-dshm\") pod \"precise-prefix-cache-test-kserve-5767855db4-jhrhr\" (UID: \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"
Jul 02 13:26:42.584250 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.584201    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-5767855db4-jhrhr\" (UID: \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"
Jul 02 13:26:42.584502 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.584251    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-tmp-dir\") pod \"precise-prefix-cache-test-kserve-5767855db4-jhrhr\" (UID: \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"
Jul 02 13:26:42.584502 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.584282    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5gsw\" (UniqueName: \"kubernetes.io/projected/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-kube-api-access-w5gsw\") pod \"precise-prefix-cache-test-kserve-5767855db4-jhrhr\" (UID: \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"
Jul 02 13:26:42.587869 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.587819    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-home\") pod \"precise-prefix-cache-test-kserve-5767855db4-jhrhr\" (UID: \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"
Jul 02 13:26:42.588123 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.588099    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-model-cache\") pod \"precise-prefix-cache-test-kserve-5767855db4-jhrhr\" (UID: \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"
Jul 02 13:26:42.588755 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.588733    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-tls-certs\") pod \"precise-prefix-cache-test-kserve-5767855db4-jhrhr\" (UID: \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"
Jul 02 13:26:42.589207 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.589187    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-5767855db4-jhrhr\" (UID: \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"
Jul 02 13:26:42.589603 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.589582    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-tmp-dir\") pod \"precise-prefix-cache-test-kserve-5767855db4-jhrhr\" (UID: \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"
Jul 02 13:26:42.592567 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.592541    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-dshm\") pod \"precise-prefix-cache-test-kserve-5767855db4-jhrhr\" (UID: \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"
Jul 02 13:26:42.597131 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.597106    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5gsw\" (UniqueName: \"kubernetes.io/projected/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-kube-api-access-w5gsw\") pod \"precise-prefix-cache-test-kserve-5767855db4-jhrhr\" (UID: \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"
Jul 02 13:26:42.743231 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.743144    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"
Jul 02 13:26:42.770722 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.770685    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8" event={"ID":"91bed416-a9bf-4dcf-af67-bceba2ece5ff","Type":"ContainerStarted","Data":"2870ef435359e39a004e1a69fd793c780176790f760b9231429926412382bf71"}
Jul 02 13:26:42.881404 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.881358    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"]
Jul 02 13:26:42.884362 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.884342    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"
Jul 02 13:26:42.887756 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.887725    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-epp-sa-dockercfg-smgbk\""
Jul 02 13:26:42.901707 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.901666    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"]
Jul 02 13:26:42.933411 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.929898    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"]
Jul 02 13:26:42.934889 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.934838    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:26:42.938221 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:26:42.938194    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a1b237e_a978_43f7_9b9b_ec8e2790bca3.slice/crio-731a74947f3d2007e6e154cdfd1469b290bfa008251eb90b6d7fb2cd24aaf255 WatchSource:0}: Error finding container 731a74947f3d2007e6e154cdfd1469b290bfa008251eb90b6d7fb2cd24aaf255: Status 404 returned error can't find the container with id 731a74947f3d2007e6e154cdfd1469b290bfa008251eb90b6d7fb2cd24aaf255
Jul 02 13:26:42.989234 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.989205    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvbkx\" (UniqueName: \"kubernetes.io/projected/f3c72b42-697d-4c59-9be7-defb3bb7016a-kube-api-access-vvbkx\") pod \"precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh\" (UID: \"f3c72b42-697d-4c59-9be7-defb3bb7016a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"
Jul 02 13:26:42.989367 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.989270    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f3c72b42-697d-4c59-9be7-defb3bb7016a-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh\" (UID: \"f3c72b42-697d-4c59-9be7-defb3bb7016a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"
Jul 02 13:26:42.989367 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.989301    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3c72b42-697d-4c59-9be7-defb3bb7016a-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh\" (UID: \"f3c72b42-697d-4c59-9be7-defb3bb7016a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"
Jul 02 13:26:42.989367 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.989347    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c72b42-697d-4c59-9be7-defb3bb7016a-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh\" (UID: \"f3c72b42-697d-4c59-9be7-defb3bb7016a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"
Jul 02 13:26:42.989367 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.989364    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c72b42-697d-4c59-9be7-defb3bb7016a-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh\" (UID: \"f3c72b42-697d-4c59-9be7-defb3bb7016a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"
Jul 02 13:26:42.989527 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:42.989421    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3c72b42-697d-4c59-9be7-defb3bb7016a-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh\" (UID: \"f3c72b42-697d-4c59-9be7-defb3bb7016a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"
Jul 02 13:26:43.090463 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:43.090427    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvbkx\" (UniqueName: \"kubernetes.io/projected/f3c72b42-697d-4c59-9be7-defb3bb7016a-kube-api-access-vvbkx\") pod \"precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh\" (UID: \"f3c72b42-697d-4c59-9be7-defb3bb7016a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"
Jul 02 13:26:43.090669 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:43.090495    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f3c72b42-697d-4c59-9be7-defb3bb7016a-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh\" (UID: \"f3c72b42-697d-4c59-9be7-defb3bb7016a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"
Jul 02 13:26:43.090669 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:43.090517    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3c72b42-697d-4c59-9be7-defb3bb7016a-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh\" (UID: \"f3c72b42-697d-4c59-9be7-defb3bb7016a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"
Jul 02 13:26:43.090669 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:43.090552    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c72b42-697d-4c59-9be7-defb3bb7016a-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh\" (UID: \"f3c72b42-697d-4c59-9be7-defb3bb7016a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"
Jul 02 13:26:43.090669 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:43.090566    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c72b42-697d-4c59-9be7-defb3bb7016a-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh\" (UID: \"f3c72b42-697d-4c59-9be7-defb3bb7016a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"
Jul 02 13:26:43.090669 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:43.090616    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3c72b42-697d-4c59-9be7-defb3bb7016a-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh\" (UID: \"f3c72b42-697d-4c59-9be7-defb3bb7016a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"
Jul 02 13:26:43.091022 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:43.090997    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f3c72b42-697d-4c59-9be7-defb3bb7016a-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh\" (UID: \"f3c72b42-697d-4c59-9be7-defb3bb7016a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"
Jul 02 13:26:43.091093 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:43.091029    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3c72b42-697d-4c59-9be7-defb3bb7016a-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh\" (UID: \"f3c72b42-697d-4c59-9be7-defb3bb7016a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"
Jul 02 13:26:43.091093 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:43.091061    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c72b42-697d-4c59-9be7-defb3bb7016a-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh\" (UID: \"f3c72b42-697d-4c59-9be7-defb3bb7016a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"
Jul 02 13:26:43.091330 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:43.091307    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3c72b42-697d-4c59-9be7-defb3bb7016a-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh\" (UID: \"f3c72b42-697d-4c59-9be7-defb3bb7016a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"
Jul 02 13:26:43.093072 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:43.093047    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c72b42-697d-4c59-9be7-defb3bb7016a-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh\" (UID: \"f3c72b42-697d-4c59-9be7-defb3bb7016a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"
Jul 02 13:26:43.099173 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:43.099150    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvbkx\" (UniqueName: \"kubernetes.io/projected/f3c72b42-697d-4c59-9be7-defb3bb7016a-kube-api-access-vvbkx\") pod \"precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh\" (UID: \"f3c72b42-697d-4c59-9be7-defb3bb7016a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"
Jul 02 13:26:43.197844 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:43.197804    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"
Jul 02 13:26:43.395950 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:43.395912    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:43.396834 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:43.396802    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8" podUID="91bed416-a9bf-4dcf-af67-bceba2ece5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Jul 02 13:26:43.409715 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:43.409596    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"]
Jul 02 13:26:43.778369 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:43.778331    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh" event={"ID":"f3c72b42-697d-4c59-9be7-defb3bb7016a","Type":"ContainerStarted","Data":"af9d081d5980b4ab2501b8641e4e66ec76ac3f1321868a235f2747bcf85ec5e1"}
Jul 02 13:26:43.778769 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:43.778416    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh" event={"ID":"f3c72b42-697d-4c59-9be7-defb3bb7016a","Type":"ContainerStarted","Data":"abe0599abaedaaa1741a872abfbee15027bc8b2bac9862584291779d62b944fc"}
Jul 02 13:26:43.780537 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:43.779997    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr" event={"ID":"2a1b237e-a978-43f7-9b9b-ec8e2790bca3","Type":"ContainerStarted","Data":"6745e8c837e9106cb318b8c0d2c2dedf1b82127ca872730acb8052d8f1081422"}
Jul 02 13:26:43.780537 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:43.780053    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr" event={"ID":"2a1b237e-a978-43f7-9b9b-ec8e2790bca3","Type":"ContainerStarted","Data":"731a74947f3d2007e6e154cdfd1469b290bfa008251eb90b6d7fb2cd24aaf255"}
Jul 02 13:26:44.396801 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:44.396761    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8" podUID="91bed416-a9bf-4dcf-af67-bceba2ece5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Jul 02 13:26:44.618863 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:44.618811    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:44.785817 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:44.785779    2583 generic.go:356] "Generic (PLEG): container finished" podID="f3c72b42-697d-4c59-9be7-defb3bb7016a" containerID="af9d081d5980b4ab2501b8641e4e66ec76ac3f1321868a235f2747bcf85ec5e1" exitCode=0
Jul 02 13:26:44.786267 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:44.785866    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh" event={"ID":"f3c72b42-697d-4c59-9be7-defb3bb7016a","Type":"ContainerDied","Data":"af9d081d5980b4ab2501b8641e4e66ec76ac3f1321868a235f2747bcf85ec5e1"}
Jul 02 13:26:44.787012 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:44.786995    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 13:26:45.396383 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:45.396342    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8" podUID="91bed416-a9bf-4dcf-af67-bceba2ece5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Jul 02 13:26:45.826463 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:45.825018    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:26:46.397410 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:46.397261    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8" podUID="91bed416-a9bf-4dcf-af67-bceba2ece5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Jul 02 13:26:47.396841 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:47.396793    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8" podUID="91bed416-a9bf-4dcf-af67-bceba2ece5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Jul 02 13:26:47.811196 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:47.811061    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh" event={"ID":"f3c72b42-697d-4c59-9be7-defb3bb7016a","Type":"ContainerStarted","Data":"9873490aeaa0c9c4b1547d5ddb267c52e82be91114e737a29ed314ff702f3355"}
Jul 02 13:26:47.813329 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:47.813298    2583 generic.go:356] "Generic (PLEG): container finished" podID="2a1b237e-a978-43f7-9b9b-ec8e2790bca3" containerID="6745e8c837e9106cb318b8c0d2c2dedf1b82127ca872730acb8052d8f1081422" exitCode=0
Jul 02 13:26:47.813549 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:47.813408    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr" event={"ID":"2a1b237e-a978-43f7-9b9b-ec8e2790bca3","Type":"ContainerDied","Data":"6745e8c837e9106cb318b8c0d2c2dedf1b82127ca872730acb8052d8f1081422"}
Jul 02 13:26:48.467356 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:48.467320    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:48.469700 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:48.469621    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-gg6n8"
Jul 02 13:26:50.834968 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:50.834929    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr" event={"ID":"2a1b237e-a978-43f7-9b9b-ec8e2790bca3","Type":"ContainerStarted","Data":"c7f887a94e254fca76749f69f7583bf901242c8768fdc1a6ba40474c14db66fe"}
Jul 02 13:26:50.859062 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:50.859001    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr" podStartSLOduration=1.86820276 podStartE2EDuration="8.858981911s" totalImagesPullingTime="2.498586497s" totalInitContainerRuntime="4.492192654s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:26:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:26:50.856188699 +0000 UTC m=+722.084099885" watchObservedRunningTime="2026-07-02 13:26:50.858981911 +0000 UTC m=+722.086893086"
Jul 02 13:26:51.743256 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:51.743217    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"
Jul 02 13:26:52.744258 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:52.744211    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"
Jul 02 13:26:52.759131 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:52.759103    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"
Jul 02 13:26:52.864582 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:52.864544    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"
Jul 02 13:26:57.030431 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:57.030401    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:26:57.876251 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:57.876213    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh_f3c72b42-697d-4c59-9be7-defb3bb7016a/main/0.log"
Jul 02 13:26:57.876694 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:57.876655    2583 generic.go:356] "Generic (PLEG): container finished" podID="f3c72b42-697d-4c59-9be7-defb3bb7016a" containerID="9873490aeaa0c9c4b1547d5ddb267c52e82be91114e737a29ed314ff702f3355" exitCode=1
Jul 02 13:26:57.876834 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:26:57.876727    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh" event={"ID":"f3c72b42-697d-4c59-9be7-defb3bb7016a","Type":"ContainerDied","Data":"9873490aeaa0c9c4b1547d5ddb267c52e82be91114e737a29ed314ff702f3355"}
Jul 02 13:27:08.152404 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:08.152339    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:27:09.916411 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:09.916353    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:27:18.946876 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:18.946845    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:27:18.999583 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:18.999556    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh_f3c72b42-697d-4c59-9be7-defb3bb7016a/main/0.log"
Jul 02 13:27:19.000031 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:19.000001    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh" event={"ID":"f3c72b42-697d-4c59-9be7-defb3bb7016a","Type":"ContainerStarted","Data":"f2fdd5be453fd2e4562dcbcbaa832c7cc87a1fbd7fe8adb1e2b7b954a84c1bcc"}
Jul 02 13:27:19.000349 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:19.000336    2583 scope.go:117] "RemoveContainer" containerID="9873490aeaa0c9c4b1547d5ddb267c52e82be91114e737a29ed314ff702f3355"
Jul 02 13:27:20.011960 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:20.011925    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh_f3c72b42-697d-4c59-9be7-defb3bb7016a/main/0.log"
Jul 02 13:27:20.012424 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:20.012365    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh" event={"ID":"f3c72b42-697d-4c59-9be7-defb3bb7016a","Type":"ContainerStarted","Data":"eee5a3f13e0e1477a15d576b2562e0b2ff644071d1a7f7728dde2ee6ceebd941"}
Jul 02 13:27:20.012626 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:20.012609    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"
Jul 02 13:27:20.041710 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:20.041651    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh" podStartSLOduration=4.058388941 podStartE2EDuration="38.041627223s" totalImagesPullingTime="33.201730378s" totalInitContainerRuntime="781.507904ms" isStatefulPod=false podCreationTimestamp="2026-07-02 13:26:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:27:20.038812686 +0000 UTC m=+751.266723863" watchObservedRunningTime="2026-07-02 13:27:20.041627223 +0000 UTC m=+751.269538399"
Jul 02 13:27:23.198339 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:23.198296    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"
Jul 02 13:27:23.198808 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:23.198402    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"
Jul 02 13:27:23.199550 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:23.199523    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh" podUID="f3c72b42-697d-4c59-9be7-defb3bb7016a" containerName="tokenizer" probeResult="failure" output="Get \"http://10.134.0.34:8082/healthz\": dial tcp 10.134.0.34:8082: connect: connection refused"
Jul 02 13:27:23.464245 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:23.464157    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:27:23.709340 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:23.709298    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:27:23.981741 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:23.981696    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:27:32.358022 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:32.357984    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:27:33.200791 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:33.200756    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"
Jul 02 13:27:33.202147 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:33.202124    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"
Jul 02 13:27:33.211060 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:27:33.211030    2583 logging.go:55] [core] [Channel #17 SubChannel #18]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.34:9003", ServerName: "10.134.0.34:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.34:9003: connect: connection refused"
Jul 02 13:27:34.199021 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:34.198974    2583 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh" podUID="f3c72b42-697d-4c59-9be7-defb3bb7016a" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.34:9003\" within 1s: context deadline exceeded"
Jul 02 13:27:54.089483 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:54.089450    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"
Jul 02 13:27:55.494168 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:55.494124    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"]
Jul 02 13:27:55.497349 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:55.497318    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"]
Jul 02 13:27:55.497659 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:55.497610    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh" podUID="f3c72b42-697d-4c59-9be7-defb3bb7016a" containerName="tokenizer" containerID="cri-o://f2fdd5be453fd2e4562dcbcbaa832c7cc87a1fbd7fe8adb1e2b7b954a84c1bcc" gracePeriod=60
Jul 02 13:27:55.507948 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:27:55.507901    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 02 13:27:55.507948 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:27:55.507948 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:27:55.507948 ip-10-0-135-143 kubenswrapper[2583]:  > containerID="eee5a3f13e0e1477a15d576b2562e0b2ff644071d1a7f7728dde2ee6ceebd941" cmd=["/bin/sleep","15"]
Jul 02 13:27:55.508191 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:55.507970    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh" message="" err=<
Jul 02 13:27:55.508191 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:27:55.508191 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:27:55.508191 ip-10-0-135-143 kubenswrapper[2583]:  >
Jul 02 13:27:55.508191 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:27:55.507988    2583 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 02 13:27:55.508191 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:27:55.508191 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:27:55.508191 ip-10-0-135-143 kubenswrapper[2583]:  > pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh" podUID="f3c72b42-697d-4c59-9be7-defb3bb7016a" containerName="main" containerID="cri-o://eee5a3f13e0e1477a15d576b2562e0b2ff644071d1a7f7728dde2ee6ceebd941"
Jul 02 13:27:55.508191 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:55.508035    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh" podUID="f3c72b42-697d-4c59-9be7-defb3bb7016a" containerName="main" containerID="cri-o://eee5a3f13e0e1477a15d576b2562e0b2ff644071d1a7f7728dde2ee6ceebd941" gracePeriod=60
Jul 02 13:27:55.517178 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:55.517132    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:27:55.609578 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:55.609541    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:27:56.199778 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:56.199750    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh_f3c72b42-697d-4c59-9be7-defb3bb7016a/main/0.log"
Jul 02 13:27:56.200135 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:56.200114    2583 generic.go:356] "Generic (PLEG): container finished" podID="f3c72b42-697d-4c59-9be7-defb3bb7016a" containerID="eee5a3f13e0e1477a15d576b2562e0b2ff644071d1a7f7728dde2ee6ceebd941" exitCode=0
Jul 02 13:27:56.200215 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:56.200191    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh" event={"ID":"f3c72b42-697d-4c59-9be7-defb3bb7016a","Type":"ContainerDied","Data":"eee5a3f13e0e1477a15d576b2562e0b2ff644071d1a7f7728dde2ee6ceebd941"}
Jul 02 13:27:56.200266 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:56.200257    2583 scope.go:117] "RemoveContainer" containerID="9873490aeaa0c9c4b1547d5ddb267c52e82be91114e737a29ed314ff702f3355"
Jul 02 13:27:57.160677 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.160652    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"
Jul 02 13:27:57.207802 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.207706    2583 generic.go:356] "Generic (PLEG): container finished" podID="f3c72b42-697d-4c59-9be7-defb3bb7016a" containerID="f2fdd5be453fd2e4562dcbcbaa832c7cc87a1fbd7fe8adb1e2b7b954a84c1bcc" exitCode=0
Jul 02 13:27:57.207802 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.207759    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"
Jul 02 13:27:57.207802 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.207785    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh" event={"ID":"f3c72b42-697d-4c59-9be7-defb3bb7016a","Type":"ContainerDied","Data":"f2fdd5be453fd2e4562dcbcbaa832c7cc87a1fbd7fe8adb1e2b7b954a84c1bcc"}
Jul 02 13:27:57.208092 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.207840    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh" event={"ID":"f3c72b42-697d-4c59-9be7-defb3bb7016a","Type":"ContainerDied","Data":"abe0599abaedaaa1741a872abfbee15027bc8b2bac9862584291779d62b944fc"}
Jul 02 13:27:57.208092 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.207864    2583 scope.go:117] "RemoveContainer" containerID="eee5a3f13e0e1477a15d576b2562e0b2ff644071d1a7f7728dde2ee6ceebd941"
Jul 02 13:27:57.216858 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.216835    2583 scope.go:117] "RemoveContainer" containerID="f2fdd5be453fd2e4562dcbcbaa832c7cc87a1fbd7fe8adb1e2b7b954a84c1bcc"
Jul 02 13:27:57.224782 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.224759    2583 scope.go:117] "RemoveContainer" containerID="af9d081d5980b4ab2501b8641e4e66ec76ac3f1321868a235f2747bcf85ec5e1"
Jul 02 13:27:57.233464 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.233413    2583 scope.go:117] "RemoveContainer" containerID="eee5a3f13e0e1477a15d576b2562e0b2ff644071d1a7f7728dde2ee6ceebd941"
Jul 02 13:27:57.233718 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:27:57.233698    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eee5a3f13e0e1477a15d576b2562e0b2ff644071d1a7f7728dde2ee6ceebd941\": container with ID starting with eee5a3f13e0e1477a15d576b2562e0b2ff644071d1a7f7728dde2ee6ceebd941 not found: ID does not exist" containerID="eee5a3f13e0e1477a15d576b2562e0b2ff644071d1a7f7728dde2ee6ceebd941"
Jul 02 13:27:57.233769 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.233728    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee5a3f13e0e1477a15d576b2562e0b2ff644071d1a7f7728dde2ee6ceebd941"} err="failed to get container status \"eee5a3f13e0e1477a15d576b2562e0b2ff644071d1a7f7728dde2ee6ceebd941\": rpc error: code = NotFound desc = could not find container \"eee5a3f13e0e1477a15d576b2562e0b2ff644071d1a7f7728dde2ee6ceebd941\": container with ID starting with eee5a3f13e0e1477a15d576b2562e0b2ff644071d1a7f7728dde2ee6ceebd941 not found: ID does not exist"
Jul 02 13:27:57.233769 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.233747    2583 scope.go:117] "RemoveContainer" containerID="f2fdd5be453fd2e4562dcbcbaa832c7cc87a1fbd7fe8adb1e2b7b954a84c1bcc"
Jul 02 13:27:57.233979 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:27:57.233961    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2fdd5be453fd2e4562dcbcbaa832c7cc87a1fbd7fe8adb1e2b7b954a84c1bcc\": container with ID starting with f2fdd5be453fd2e4562dcbcbaa832c7cc87a1fbd7fe8adb1e2b7b954a84c1bcc not found: ID does not exist" containerID="f2fdd5be453fd2e4562dcbcbaa832c7cc87a1fbd7fe8adb1e2b7b954a84c1bcc"
Jul 02 13:27:57.234021 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.233985    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2fdd5be453fd2e4562dcbcbaa832c7cc87a1fbd7fe8adb1e2b7b954a84c1bcc"} err="failed to get container status \"f2fdd5be453fd2e4562dcbcbaa832c7cc87a1fbd7fe8adb1e2b7b954a84c1bcc\": rpc error: code = NotFound desc = could not find container \"f2fdd5be453fd2e4562dcbcbaa832c7cc87a1fbd7fe8adb1e2b7b954a84c1bcc\": container with ID starting with f2fdd5be453fd2e4562dcbcbaa832c7cc87a1fbd7fe8adb1e2b7b954a84c1bcc not found: ID does not exist"
Jul 02 13:27:57.234021 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.234002    2583 scope.go:117] "RemoveContainer" containerID="af9d081d5980b4ab2501b8641e4e66ec76ac3f1321868a235f2747bcf85ec5e1"
Jul 02 13:27:57.234247 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:27:57.234225    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af9d081d5980b4ab2501b8641e4e66ec76ac3f1321868a235f2747bcf85ec5e1\": container with ID starting with af9d081d5980b4ab2501b8641e4e66ec76ac3f1321868a235f2747bcf85ec5e1 not found: ID does not exist" containerID="af9d081d5980b4ab2501b8641e4e66ec76ac3f1321868a235f2747bcf85ec5e1"
Jul 02 13:27:57.234292 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.234258    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af9d081d5980b4ab2501b8641e4e66ec76ac3f1321868a235f2747bcf85ec5e1"} err="failed to get container status \"af9d081d5980b4ab2501b8641e4e66ec76ac3f1321868a235f2747bcf85ec5e1\": rpc error: code = NotFound desc = could not find container \"af9d081d5980b4ab2501b8641e4e66ec76ac3f1321868a235f2747bcf85ec5e1\": container with ID starting with af9d081d5980b4ab2501b8641e4e66ec76ac3f1321868a235f2747bcf85ec5e1 not found: ID does not exist"
Jul 02 13:27:57.247970 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.247947    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3c72b42-697d-4c59-9be7-defb3bb7016a-kserve-provision-location\") pod \"f3c72b42-697d-4c59-9be7-defb3bb7016a\" (UID: \"f3c72b42-697d-4c59-9be7-defb3bb7016a\") "
Jul 02 13:27:57.248071 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.248055    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c72b42-697d-4c59-9be7-defb3bb7016a-tls-certs\") pod \"f3c72b42-697d-4c59-9be7-defb3bb7016a\" (UID: \"f3c72b42-697d-4c59-9be7-defb3bb7016a\") "
Jul 02 13:27:57.248110 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.248088    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3c72b42-697d-4c59-9be7-defb3bb7016a-tokenizer-tmp\") pod \"f3c72b42-697d-4c59-9be7-defb3bb7016a\" (UID: \"f3c72b42-697d-4c59-9be7-defb3bb7016a\") "
Jul 02 13:27:57.248160 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.248114    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f3c72b42-697d-4c59-9be7-defb3bb7016a-tokenizer-uds\") pod \"f3c72b42-697d-4c59-9be7-defb3bb7016a\" (UID: \"f3c72b42-697d-4c59-9be7-defb3bb7016a\") "
Jul 02 13:27:57.248160 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.248136    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c72b42-697d-4c59-9be7-defb3bb7016a-tokenizer-cache\") pod \"f3c72b42-697d-4c59-9be7-defb3bb7016a\" (UID: \"f3c72b42-697d-4c59-9be7-defb3bb7016a\") "
Jul 02 13:27:57.248267 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.248167    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvbkx\" (UniqueName: \"kubernetes.io/projected/f3c72b42-697d-4c59-9be7-defb3bb7016a-kube-api-access-vvbkx\") pod \"f3c72b42-697d-4c59-9be7-defb3bb7016a\" (UID: \"f3c72b42-697d-4c59-9be7-defb3bb7016a\") "
Jul 02 13:27:57.248379 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.248356    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c72b42-697d-4c59-9be7-defb3bb7016a-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "f3c72b42-697d-4c59-9be7-defb3bb7016a" (UID: "f3c72b42-697d-4c59-9be7-defb3bb7016a"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:27:57.248501 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.248465    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c72b42-697d-4c59-9be7-defb3bb7016a-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "f3c72b42-697d-4c59-9be7-defb3bb7016a" (UID: "f3c72b42-697d-4c59-9be7-defb3bb7016a"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:27:57.248588 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.248548    2583 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3c72b42-697d-4c59-9be7-defb3bb7016a-tokenizer-tmp\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:27:57.248588 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.248577    2583 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f3c72b42-697d-4c59-9be7-defb3bb7016a-tokenizer-uds\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:27:57.248588 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.248550    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c72b42-697d-4c59-9be7-defb3bb7016a-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "f3c72b42-697d-4c59-9be7-defb3bb7016a" (UID: "f3c72b42-697d-4c59-9be7-defb3bb7016a"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:27:57.248783 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.248765    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c72b42-697d-4c59-9be7-defb3bb7016a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f3c72b42-697d-4c59-9be7-defb3bb7016a" (UID: "f3c72b42-697d-4c59-9be7-defb3bb7016a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:27:57.250303 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.250281    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3c72b42-697d-4c59-9be7-defb3bb7016a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3c72b42-697d-4c59-9be7-defb3bb7016a" (UID: "f3c72b42-697d-4c59-9be7-defb3bb7016a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:27:57.250471 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.250447    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3c72b42-697d-4c59-9be7-defb3bb7016a-kube-api-access-vvbkx" (OuterVolumeSpecName: "kube-api-access-vvbkx") pod "f3c72b42-697d-4c59-9be7-defb3bb7016a" (UID: "f3c72b42-697d-4c59-9be7-defb3bb7016a"). InnerVolumeSpecName "kube-api-access-vvbkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:27:57.349760 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.349726    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3c72b42-697d-4c59-9be7-defb3bb7016a-kserve-provision-location\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:27:57.349760 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.349759    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c72b42-697d-4c59-9be7-defb3bb7016a-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:27:57.349760 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.349769    2583 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c72b42-697d-4c59-9be7-defb3bb7016a-tokenizer-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:27:57.349999 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.349778    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvbkx\" (UniqueName: \"kubernetes.io/projected/f3c72b42-697d-4c59-9be7-defb3bb7016a-kube-api-access-vvbkx\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:27:57.535440 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.535407    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"]
Jul 02 13:27:57.540002 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:57.539973    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-cd9b9878xzrmh"]
Jul 02 13:27:59.234013 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:27:59.233981    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3c72b42-697d-4c59-9be7-defb3bb7016a" path="/var/lib/kubelet/pods/f3c72b42-697d-4c59-9be7-defb3bb7016a/volumes"
Jul 02 13:28:10.517979 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:10.517914    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr" podUID="2a1b237e-a978-43f7-9b9b-ec8e2790bca3" containerName="main" containerID="cri-o://c7f887a94e254fca76749f69f7583bf901242c8768fdc1a6ba40474c14db66fe" gracePeriod=45
Jul 02 13:28:10.766601 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:10.766574    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"
Jul 02 13:28:10.873247 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:10.873216    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-tmp-dir\") pod \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\" (UID: \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\") "
Jul 02 13:28:10.873484 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:10.873264    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5gsw\" (UniqueName: \"kubernetes.io/projected/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-kube-api-access-w5gsw\") pod \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\" (UID: \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\") "
Jul 02 13:28:10.873484 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:10.873289    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-model-cache\") pod \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\" (UID: \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\") "
Jul 02 13:28:10.873484 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:10.873444    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-kserve-provision-location\") pod \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\" (UID: \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\") "
Jul 02 13:28:10.873682 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:10.873492    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-dshm\") pod \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\" (UID: \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\") "
Jul 02 13:28:10.873682 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:10.873539    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-tls-certs\") pod \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\" (UID: \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\") "
Jul 02 13:28:10.873682 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:10.873564    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-home\") pod \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\" (UID: \"2a1b237e-a978-43f7-9b9b-ec8e2790bca3\") "
Jul 02 13:28:10.873682 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:10.873593    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2a1b237e-a978-43f7-9b9b-ec8e2790bca3" (UID: "2a1b237e-a978-43f7-9b9b-ec8e2790bca3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:28:10.873682 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:10.873607    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-model-cache" (OuterVolumeSpecName: "model-cache") pod "2a1b237e-a978-43f7-9b9b-ec8e2790bca3" (UID: "2a1b237e-a978-43f7-9b9b-ec8e2790bca3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:28:10.873909 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:10.873886    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-home" (OuterVolumeSpecName: "home") pod "2a1b237e-a978-43f7-9b9b-ec8e2790bca3" (UID: "2a1b237e-a978-43f7-9b9b-ec8e2790bca3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:28:10.873971 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:10.873908    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-tmp-dir\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:28:10.873971 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:10.873935    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:28:10.875505 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:10.875475    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-kube-api-access-w5gsw" (OuterVolumeSpecName: "kube-api-access-w5gsw") pod "2a1b237e-a978-43f7-9b9b-ec8e2790bca3" (UID: "2a1b237e-a978-43f7-9b9b-ec8e2790bca3"). InnerVolumeSpecName "kube-api-access-w5gsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:28:10.875621 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:10.875584    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-dshm" (OuterVolumeSpecName: "dshm") pod "2a1b237e-a978-43f7-9b9b-ec8e2790bca3" (UID: "2a1b237e-a978-43f7-9b9b-ec8e2790bca3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:28:10.875665 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:10.875618    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a1b237e-a978-43f7-9b9b-ec8e2790bca3" (UID: "2a1b237e-a978-43f7-9b9b-ec8e2790bca3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:28:10.936092 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:10.936038    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2a1b237e-a978-43f7-9b9b-ec8e2790bca3" (UID: "2a1b237e-a978-43f7-9b9b-ec8e2790bca3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:28:10.975147 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:10.975111    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-kserve-provision-location\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:28:10.975147 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:10.975146    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-dshm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:28:10.975286 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:10.975156    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:28:10.975286 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:10.975166    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-home\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:28:10.975286 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:10.975174    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5gsw\" (UniqueName: \"kubernetes.io/projected/2a1b237e-a978-43f7-9b9b-ec8e2790bca3-kube-api-access-w5gsw\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:28:11.277328 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:11.277288    2583 generic.go:356] "Generic (PLEG): container finished" podID="2a1b237e-a978-43f7-9b9b-ec8e2790bca3" containerID="c7f887a94e254fca76749f69f7583bf901242c8768fdc1a6ba40474c14db66fe" exitCode=0
Jul 02 13:28:11.277548 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:11.277335    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr" event={"ID":"2a1b237e-a978-43f7-9b9b-ec8e2790bca3","Type":"ContainerDied","Data":"c7f887a94e254fca76749f69f7583bf901242c8768fdc1a6ba40474c14db66fe"}
Jul 02 13:28:11.277548 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:11.277363    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"
Jul 02 13:28:11.277548 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:11.277425    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr" event={"ID":"2a1b237e-a978-43f7-9b9b-ec8e2790bca3","Type":"ContainerDied","Data":"731a74947f3d2007e6e154cdfd1469b290bfa008251eb90b6d7fb2cd24aaf255"}
Jul 02 13:28:11.277548 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:11.277456    2583 scope.go:117] "RemoveContainer" containerID="c7f887a94e254fca76749f69f7583bf901242c8768fdc1a6ba40474c14db66fe"
Jul 02 13:28:11.285981 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:11.285960    2583 scope.go:117] "RemoveContainer" containerID="6745e8c837e9106cb318b8c0d2c2dedf1b82127ca872730acb8052d8f1081422"
Jul 02 13:28:11.294555 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:11.294532    2583 scope.go:117] "RemoveContainer" containerID="c7f887a94e254fca76749f69f7583bf901242c8768fdc1a6ba40474c14db66fe"
Jul 02 13:28:11.294958 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:28:11.294941    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7f887a94e254fca76749f69f7583bf901242c8768fdc1a6ba40474c14db66fe\": container with ID starting with c7f887a94e254fca76749f69f7583bf901242c8768fdc1a6ba40474c14db66fe not found: ID does not exist" containerID="c7f887a94e254fca76749f69f7583bf901242c8768fdc1a6ba40474c14db66fe"
Jul 02 13:28:11.295005 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:11.294967    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7f887a94e254fca76749f69f7583bf901242c8768fdc1a6ba40474c14db66fe"} err="failed to get container status \"c7f887a94e254fca76749f69f7583bf901242c8768fdc1a6ba40474c14db66fe\": rpc error: code = NotFound desc = could not find container \"c7f887a94e254fca76749f69f7583bf901242c8768fdc1a6ba40474c14db66fe\": container with ID starting with c7f887a94e254fca76749f69f7583bf901242c8768fdc1a6ba40474c14db66fe not found: ID does not exist"
Jul 02 13:28:11.295005 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:11.294983    2583 scope.go:117] "RemoveContainer" containerID="6745e8c837e9106cb318b8c0d2c2dedf1b82127ca872730acb8052d8f1081422"
Jul 02 13:28:11.295233 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:28:11.295216    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6745e8c837e9106cb318b8c0d2c2dedf1b82127ca872730acb8052d8f1081422\": container with ID starting with 6745e8c837e9106cb318b8c0d2c2dedf1b82127ca872730acb8052d8f1081422 not found: ID does not exist" containerID="6745e8c837e9106cb318b8c0d2c2dedf1b82127ca872730acb8052d8f1081422"
Jul 02 13:28:11.295290 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:11.295238    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6745e8c837e9106cb318b8c0d2c2dedf1b82127ca872730acb8052d8f1081422"} err="failed to get container status \"6745e8c837e9106cb318b8c0d2c2dedf1b82127ca872730acb8052d8f1081422\": rpc error: code = NotFound desc = could not find container \"6745e8c837e9106cb318b8c0d2c2dedf1b82127ca872730acb8052d8f1081422\": container with ID starting with 6745e8c837e9106cb318b8c0d2c2dedf1b82127ca872730acb8052d8f1081422 not found: ID does not exist"
Jul 02 13:28:11.323129 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:11.323093    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"]
Jul 02 13:28:11.326585 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:11.326561    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5767855db4-jhrhr"]
Jul 02 13:28:13.234261 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:13.234224    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a1b237e-a978-43f7-9b9b-ec8e2790bca3" path="/var/lib/kubelet/pods/2a1b237e-a978-43f7-9b9b-ec8e2790bca3/volumes"
Jul 02 13:28:19.399651 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.399615    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9"]
Jul 02 13:28:19.400050 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.399994    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a1b237e-a978-43f7-9b9b-ec8e2790bca3" containerName="storage-initializer"
Jul 02 13:28:19.400050 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.400005    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a1b237e-a978-43f7-9b9b-ec8e2790bca3" containerName="storage-initializer"
Jul 02 13:28:19.400050 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.400019    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3c72b42-697d-4c59-9be7-defb3bb7016a" containerName="main"
Jul 02 13:28:19.400050 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.400024    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3c72b42-697d-4c59-9be7-defb3bb7016a" containerName="main"
Jul 02 13:28:19.400050 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.400033    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3c72b42-697d-4c59-9be7-defb3bb7016a" containerName="storage-initializer"
Jul 02 13:28:19.400050 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.400039    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3c72b42-697d-4c59-9be7-defb3bb7016a" containerName="storage-initializer"
Jul 02 13:28:19.400050 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.400046    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3c72b42-697d-4c59-9be7-defb3bb7016a" containerName="main"
Jul 02 13:28:19.400050 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.400053    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3c72b42-697d-4c59-9be7-defb3bb7016a" containerName="main"
Jul 02 13:28:19.400320 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.400059    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3c72b42-697d-4c59-9be7-defb3bb7016a" containerName="tokenizer"
Jul 02 13:28:19.400320 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.400065    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3c72b42-697d-4c59-9be7-defb3bb7016a" containerName="tokenizer"
Jul 02 13:28:19.400320 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.400083    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a1b237e-a978-43f7-9b9b-ec8e2790bca3" containerName="main"
Jul 02 13:28:19.400320 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.400088    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a1b237e-a978-43f7-9b9b-ec8e2790bca3" containerName="main"
Jul 02 13:28:19.400320 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.400141    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3c72b42-697d-4c59-9be7-defb3bb7016a" containerName="main"
Jul 02 13:28:19.400320 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.400152    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3c72b42-697d-4c59-9be7-defb3bb7016a" containerName="tokenizer"
Jul 02 13:28:19.400320 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.400158    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a1b237e-a978-43f7-9b9b-ec8e2790bca3" containerName="main"
Jul 02 13:28:19.402151 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.402134    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9"
Jul 02 13:28:19.406102 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.406081    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7s8sh\""
Jul 02 13:28:19.406334 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.406320    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 02 13:28:19.413848 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.413823    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9"]
Jul 02 13:28:19.423231 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.423201    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:28:19.553404 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.553364    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9\" (UID: \"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9"
Jul 02 13:28:19.553551 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.553435    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9\" (UID: \"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9"
Jul 02 13:28:19.553551 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.553531    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9\" (UID: \"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9"
Jul 02 13:28:19.553642 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.553571    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9\" (UID: \"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9"
Jul 02 13:28:19.553642 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.553590    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r5bk\" (UniqueName: \"kubernetes.io/projected/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-kube-api-access-7r5bk\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9\" (UID: \"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9"
Jul 02 13:28:19.553710 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.553677    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9\" (UID: \"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9"
Jul 02 13:28:19.654657 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.654558    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9\" (UID: \"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9"
Jul 02 13:28:19.654657 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.654623    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9\" (UID: \"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9"
Jul 02 13:28:19.654657 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.654643    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9\" (UID: \"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9"
Jul 02 13:28:19.654960 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.654663    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r5bk\" (UniqueName: \"kubernetes.io/projected/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-kube-api-access-7r5bk\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9\" (UID: \"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9"
Jul 02 13:28:19.654960 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.654707    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9\" (UID: \"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9"
Jul 02 13:28:19.654960 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.654733    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9\" (UID: \"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9"
Jul 02 13:28:19.655124 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.655035    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9\" (UID: \"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9"
Jul 02 13:28:19.655124 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.655061    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9\" (UID: \"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9"
Jul 02 13:28:19.655124 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.655106    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9\" (UID: \"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9"
Jul 02 13:28:19.656968 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.656944    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9\" (UID: \"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9"
Jul 02 13:28:19.657120 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.657102    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9\" (UID: \"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9"
Jul 02 13:28:19.663307 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.663275    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r5bk\" (UniqueName: \"kubernetes.io/projected/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-kube-api-access-7r5bk\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9\" (UID: \"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9"
Jul 02 13:28:19.712889 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.712823    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9"
Jul 02 13:28:19.724231 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.724205    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn"]
Jul 02 13:28:19.724739 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.724722    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3c72b42-697d-4c59-9be7-defb3bb7016a" containerName="main"
Jul 02 13:28:19.726949 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.726925    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn"
Jul 02 13:28:19.729902 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.729880    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-sr6d9\""
Jul 02 13:28:19.747599 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.747569    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn"]
Jul 02 13:28:19.776104 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.776066    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:28:19.856015 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.855914    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9"]
Jul 02 13:28:19.857003 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.856977    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp4xb\" (UniqueName: \"kubernetes.io/projected/843c731d-702b-439a-bf06-72244f3680dc-kube-api-access-kp4xb\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn\" (UID: \"843c731d-702b-439a-bf06-72244f3680dc\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn"
Jul 02 13:28:19.857078 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.857024    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/843c731d-702b-439a-bf06-72244f3680dc-tokenizer-uds\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn\" (UID: \"843c731d-702b-439a-bf06-72244f3680dc\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn"
Jul 02 13:28:19.857128 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.857113    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/843c731d-702b-439a-bf06-72244f3680dc-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn\" (UID: \"843c731d-702b-439a-bf06-72244f3680dc\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn"
Jul 02 13:28:19.858402 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:28:19.858364    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dbf1bf8_b4fa_432f_90b6_ae230cb27ad0.slice/crio-18e053a63895b4ec092c80ddfde5d444275e27a18226f3eda557e8c2f25035cf WatchSource:0}: Error finding container 18e053a63895b4ec092c80ddfde5d444275e27a18226f3eda557e8c2f25035cf: Status 404 returned error can't find the container with id 18e053a63895b4ec092c80ddfde5d444275e27a18226f3eda557e8c2f25035cf
Jul 02 13:28:19.958574 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.958535    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp4xb\" (UniqueName: \"kubernetes.io/projected/843c731d-702b-439a-bf06-72244f3680dc-kube-api-access-kp4xb\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn\" (UID: \"843c731d-702b-439a-bf06-72244f3680dc\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn"
Jul 02 13:28:19.958761 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.958585    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/843c731d-702b-439a-bf06-72244f3680dc-tokenizer-uds\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn\" (UID: \"843c731d-702b-439a-bf06-72244f3680dc\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn"
Jul 02 13:28:19.958761 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.958641    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/843c731d-702b-439a-bf06-72244f3680dc-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn\" (UID: \"843c731d-702b-439a-bf06-72244f3680dc\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn"
Jul 02 13:28:19.959038 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.959013    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/843c731d-702b-439a-bf06-72244f3680dc-tokenizer-uds\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn\" (UID: \"843c731d-702b-439a-bf06-72244f3680dc\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn"
Jul 02 13:28:19.961254 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.961233    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/843c731d-702b-439a-bf06-72244f3680dc-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn\" (UID: \"843c731d-702b-439a-bf06-72244f3680dc\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn"
Jul 02 13:28:19.967054 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:19.967028    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp4xb\" (UniqueName: \"kubernetes.io/projected/843c731d-702b-439a-bf06-72244f3680dc-kube-api-access-kp4xb\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn\" (UID: \"843c731d-702b-439a-bf06-72244f3680dc\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn"
Jul 02 13:28:20.049097 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:20.049052    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn"
Jul 02 13:28:20.187011 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:20.186979    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn"]
Jul 02 13:28:20.189463 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:28:20.189438    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod843c731d_702b_439a_bf06_72244f3680dc.slice/crio-271e39bb82f9d7246756136c9610718ad1dde5536dc318257cfd071354aa770f WatchSource:0}: Error finding container 271e39bb82f9d7246756136c9610718ad1dde5536dc318257cfd071354aa770f: Status 404 returned error can't find the container with id 271e39bb82f9d7246756136c9610718ad1dde5536dc318257cfd071354aa770f
Jul 02 13:28:20.326018 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:20.325970    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9" event={"ID":"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0","Type":"ContainerStarted","Data":"aa2a21645e062e52f01692261cd933822dbdac1702cc9508e98d34dae17d9418"}
Jul 02 13:28:20.326193 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:20.326049    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9" event={"ID":"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0","Type":"ContainerStarted","Data":"18e053a63895b4ec092c80ddfde5d444275e27a18226f3eda557e8c2f25035cf"}
Jul 02 13:28:20.327639 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:20.327615    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn" event={"ID":"843c731d-702b-439a-bf06-72244f3680dc","Type":"ContainerStarted","Data":"3273c649113134896847d8e091fcd014e3002db71d796bfa5161045ce4ce4886"}
Jul 02 13:28:20.327766 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:20.327653    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn" event={"ID":"843c731d-702b-439a-bf06-72244f3680dc","Type":"ContainerStarted","Data":"271e39bb82f9d7246756136c9610718ad1dde5536dc318257cfd071354aa770f"}
Jul 02 13:28:20.327766 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:20.327671    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn"
Jul 02 13:28:20.347732 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:20.347667    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9" podStartSLOduration=1.347645448 podStartE2EDuration="1.347645448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:28:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:28:20.347286815 +0000 UTC m=+811.575197989" watchObservedRunningTime="2026-07-02 13:28:20.347645448 +0000 UTC m=+811.575556625"
Jul 02 13:28:20.373883 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:20.373810    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn" podStartSLOduration=1.3737853819999999 podStartE2EDuration="1.373785382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:28:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:28:20.371847099 +0000 UTC m=+811.599758287" watchObservedRunningTime="2026-07-02 13:28:20.373785382 +0000 UTC m=+811.601696561"
Jul 02 13:28:20.713412 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:20.713336    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9"
Jul 02 13:28:29.713810 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:29.713763    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9"
Jul 02 13:28:29.726248 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:29.726213    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9"
Jul 02 13:28:30.387289 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:30.387256    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9"
Jul 02 13:28:34.923233 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:34.923194    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:28:51.334667 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:51.334634    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn"
Jul 02 13:28:52.698858 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:52.698827    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:28:52.703149 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:52.703116    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9"]
Jul 02 13:28:52.708047 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:52.708019    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn"]
Jul 02 13:28:52.717202 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:28:52.717149    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 02 13:28:52.717202 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:28:52.717202 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:28:52.717202 ip-10-0-135-143 kubenswrapper[2583]:  > containerID="3273c649113134896847d8e091fcd014e3002db71d796bfa5161045ce4ce4886" cmd=["/bin/sleep","15"]
Jul 02 13:28:52.717493 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:52.717223    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn" message="" err=<
Jul 02 13:28:52.717493 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:28:52.717493 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:28:52.717493 ip-10-0-135-143 kubenswrapper[2583]:  >
Jul 02 13:28:52.717493 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:28:52.717241    2583 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 02 13:28:52.717493 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:28:52.717493 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:28:52.717493 ip-10-0-135-143 kubenswrapper[2583]:  > pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn" podUID="843c731d-702b-439a-bf06-72244f3680dc" containerName="main" containerID="cri-o://3273c649113134896847d8e091fcd014e3002db71d796bfa5161045ce4ce4886"
Jul 02 13:28:52.717493 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:52.717287    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn" podUID="843c731d-702b-439a-bf06-72244f3680dc" containerName="main" containerID="cri-o://3273c649113134896847d8e091fcd014e3002db71d796bfa5161045ce4ce4886" gracePeriod=60
Jul 02 13:28:52.760274 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:52.760245    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:28:52.970610 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:52.970583    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn"
Jul 02 13:28:53.066926 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:53.066891    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/843c731d-702b-439a-bf06-72244f3680dc-tls-certs\") pod \"843c731d-702b-439a-bf06-72244f3680dc\" (UID: \"843c731d-702b-439a-bf06-72244f3680dc\") "
Jul 02 13:28:53.067103 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:53.066946    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/843c731d-702b-439a-bf06-72244f3680dc-tokenizer-uds\") pod \"843c731d-702b-439a-bf06-72244f3680dc\" (UID: \"843c731d-702b-439a-bf06-72244f3680dc\") "
Jul 02 13:28:53.067103 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:53.066976    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kp4xb\" (UniqueName: \"kubernetes.io/projected/843c731d-702b-439a-bf06-72244f3680dc-kube-api-access-kp4xb\") pod \"843c731d-702b-439a-bf06-72244f3680dc\" (UID: \"843c731d-702b-439a-bf06-72244f3680dc\") "
Jul 02 13:28:53.067249 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:53.067226    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/843c731d-702b-439a-bf06-72244f3680dc-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "843c731d-702b-439a-bf06-72244f3680dc" (UID: "843c731d-702b-439a-bf06-72244f3680dc"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:28:53.069090 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:53.069057    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/843c731d-702b-439a-bf06-72244f3680dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "843c731d-702b-439a-bf06-72244f3680dc" (UID: "843c731d-702b-439a-bf06-72244f3680dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:28:53.069210 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:53.069100    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/843c731d-702b-439a-bf06-72244f3680dc-kube-api-access-kp4xb" (OuterVolumeSpecName: "kube-api-access-kp4xb") pod "843c731d-702b-439a-bf06-72244f3680dc" (UID: "843c731d-702b-439a-bf06-72244f3680dc"). InnerVolumeSpecName "kube-api-access-kp4xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:28:53.168471 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:53.168432    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/843c731d-702b-439a-bf06-72244f3680dc-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:28:53.168471 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:53.168463    2583 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/843c731d-702b-439a-bf06-72244f3680dc-tokenizer-uds\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:28:53.168471 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:53.168473    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp4xb\" (UniqueName: \"kubernetes.io/projected/843c731d-702b-439a-bf06-72244f3680dc-kube-api-access-kp4xb\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:28:53.502238 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:53.502141    2583 generic.go:356] "Generic (PLEG): container finished" podID="843c731d-702b-439a-bf06-72244f3680dc" containerID="3273c649113134896847d8e091fcd014e3002db71d796bfa5161045ce4ce4886" exitCode=0
Jul 02 13:28:53.502238 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:53.502198    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn"
Jul 02 13:28:53.502238 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:53.502218    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn" event={"ID":"843c731d-702b-439a-bf06-72244f3680dc","Type":"ContainerDied","Data":"3273c649113134896847d8e091fcd014e3002db71d796bfa5161045ce4ce4886"}
Jul 02 13:28:53.502539 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:53.502291    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn" event={"ID":"843c731d-702b-439a-bf06-72244f3680dc","Type":"ContainerDied","Data":"271e39bb82f9d7246756136c9610718ad1dde5536dc318257cfd071354aa770f"}
Jul 02 13:28:53.502539 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:53.502312    2583 scope.go:117] "RemoveContainer" containerID="3273c649113134896847d8e091fcd014e3002db71d796bfa5161045ce4ce4886"
Jul 02 13:28:53.511330 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:53.511303    2583 scope.go:117] "RemoveContainer" containerID="3273c649113134896847d8e091fcd014e3002db71d796bfa5161045ce4ce4886"
Jul 02 13:28:53.511608 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:28:53.511588    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3273c649113134896847d8e091fcd014e3002db71d796bfa5161045ce4ce4886\": container with ID starting with 3273c649113134896847d8e091fcd014e3002db71d796bfa5161045ce4ce4886 not found: ID does not exist" containerID="3273c649113134896847d8e091fcd014e3002db71d796bfa5161045ce4ce4886"
Jul 02 13:28:53.511684 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:53.511614    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3273c649113134896847d8e091fcd014e3002db71d796bfa5161045ce4ce4886"} err="failed to get container status \"3273c649113134896847d8e091fcd014e3002db71d796bfa5161045ce4ce4886\": rpc error: code = NotFound desc = could not find container \"3273c649113134896847d8e091fcd014e3002db71d796bfa5161045ce4ce4886\": container with ID starting with 3273c649113134896847d8e091fcd014e3002db71d796bfa5161045ce4ce4886 not found: ID does not exist"
Jul 02 13:28:53.520291 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:53.520261    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn"]
Jul 02 13:28:53.526668 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:53.526641    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schefm4qn"]
Jul 02 13:28:55.233713 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:28:55.233679    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="843c731d-702b-439a-bf06-72244f3680dc" path="/var/lib/kubelet/pods/843c731d-702b-439a-bf06-72244f3680dc/volumes"
Jul 02 13:29:07.726164 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:07.726098    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9" podUID="2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0" containerName="main" containerID="cri-o://aa2a21645e062e52f01692261cd933822dbdac1702cc9508e98d34dae17d9418" gracePeriod=45
Jul 02 13:29:07.997839 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:07.997812    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9"
Jul 02 13:29:08.107097 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:08.107060    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-tls-certs\") pod \"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0\" (UID: \"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0\") "
Jul 02 13:29:08.107290 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:08.107138    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7r5bk\" (UniqueName: \"kubernetes.io/projected/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-kube-api-access-7r5bk\") pod \"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0\" (UID: \"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0\") "
Jul 02 13:29:08.107290 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:08.107165    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-tmp-dir\") pod \"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0\" (UID: \"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0\") "
Jul 02 13:29:08.107290 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:08.107193    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-dshm\") pod \"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0\" (UID: \"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0\") "
Jul 02 13:29:08.107290 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:08.107234    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-home\") pod \"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0\" (UID: \"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0\") "
Jul 02 13:29:08.107290 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:08.107279    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-model-cache\") pod \"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0\" (UID: \"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0\") "
Jul 02 13:29:08.107581 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:08.107528    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0" (UID: "2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:29:08.107581 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:08.107545    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-home" (OuterVolumeSpecName: "home") pod "2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0" (UID: "2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:29:08.107677 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:08.107645    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-model-cache" (OuterVolumeSpecName: "model-cache") pod "2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0" (UID: "2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:29:08.107757 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:08.107735    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-tmp-dir\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:29:08.107809 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:08.107761    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-home\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:29:08.109317 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:08.109292    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-dshm" (OuterVolumeSpecName: "dshm") pod "2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0" (UID: "2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:29:08.109317 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:08.109302    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-kube-api-access-7r5bk" (OuterVolumeSpecName: "kube-api-access-7r5bk") pod "2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0" (UID: "2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0"). InnerVolumeSpecName "kube-api-access-7r5bk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:29:08.109487 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:08.109327    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0" (UID: "2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:29:08.209221 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:08.209181    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:29:08.209221 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:08.209216    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r5bk\" (UniqueName: \"kubernetes.io/projected/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-kube-api-access-7r5bk\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:29:08.209221 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:08.209226    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-dshm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:29:08.209495 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:08.209237    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:29:08.577059 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:08.577020    2583 generic.go:356] "Generic (PLEG): container finished" podID="2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0" containerID="aa2a21645e062e52f01692261cd933822dbdac1702cc9508e98d34dae17d9418" exitCode=0
Jul 02 13:29:08.577236 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:08.577070    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9"
Jul 02 13:29:08.577236 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:08.577105    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9" event={"ID":"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0","Type":"ContainerDied","Data":"aa2a21645e062e52f01692261cd933822dbdac1702cc9508e98d34dae17d9418"}
Jul 02 13:29:08.577236 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:08.577168    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9" event={"ID":"2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0","Type":"ContainerDied","Data":"18e053a63895b4ec092c80ddfde5d444275e27a18226f3eda557e8c2f25035cf"}
Jul 02 13:29:08.577236 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:08.577193    2583 scope.go:117] "RemoveContainer" containerID="aa2a21645e062e52f01692261cd933822dbdac1702cc9508e98d34dae17d9418"
Jul 02 13:29:08.586467 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:08.586443    2583 scope.go:117] "RemoveContainer" containerID="aa2a21645e062e52f01692261cd933822dbdac1702cc9508e98d34dae17d9418"
Jul 02 13:29:08.586767 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:29:08.586746    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa2a21645e062e52f01692261cd933822dbdac1702cc9508e98d34dae17d9418\": container with ID starting with aa2a21645e062e52f01692261cd933822dbdac1702cc9508e98d34dae17d9418 not found: ID does not exist" containerID="aa2a21645e062e52f01692261cd933822dbdac1702cc9508e98d34dae17d9418"
Jul 02 13:29:08.586815 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:08.586776    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa2a21645e062e52f01692261cd933822dbdac1702cc9508e98d34dae17d9418"} err="failed to get container status \"aa2a21645e062e52f01692261cd933822dbdac1702cc9508e98d34dae17d9418\": rpc error: code = NotFound desc = could not find container \"aa2a21645e062e52f01692261cd933822dbdac1702cc9508e98d34dae17d9418\": container with ID starting with aa2a21645e062e52f01692261cd933822dbdac1702cc9508e98d34dae17d9418 not found: ID does not exist"
Jul 02 13:29:08.600796 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:08.600768    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9"]
Jul 02 13:29:08.607038 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:08.607011    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879m7jg9"]
Jul 02 13:29:09.233997 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:09.233961    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0" path="/var/lib/kubelet/pods/2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0/volumes"
Jul 02 13:29:21.998170 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:21.998137    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh"]
Jul 02 13:29:21.998722 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:21.998694    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="843c731d-702b-439a-bf06-72244f3680dc" containerName="main"
Jul 02 13:29:21.998722 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:21.998717    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="843c731d-702b-439a-bf06-72244f3680dc" containerName="main"
Jul 02 13:29:21.998900 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:21.998744    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0" containerName="main"
Jul 02 13:29:21.998900 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:21.998755    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0" containerName="main"
Jul 02 13:29:21.998900 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:21.998862    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="843c731d-702b-439a-bf06-72244f3680dc" containerName="main"
Jul 02 13:29:21.998900 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:21.998873    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dbf1bf8-b4fa-432f-90b6-ae230cb27ad0" containerName="main"
Jul 02 13:29:22.001975 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:22.001959    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh"
Jul 02 13:29:22.005862 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:22.005839    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 02 13:29:22.005981 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:22.005860    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7s8sh\""
Jul 02 13:29:22.014485 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:22.014458    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh"]
Jul 02 13:29:22.024169 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:22.024138    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:29:22.035382 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:22.035351    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vd7h\" (UniqueName: \"kubernetes.io/projected/019c88cd-243c-4137-af28-55e47b730d1f-kube-api-access-9vd7h\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh\" (UID: \"019c88cd-243c-4137-af28-55e47b730d1f\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh"
Jul 02 13:29:22.035544 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:22.035463    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/019c88cd-243c-4137-af28-55e47b730d1f-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh\" (UID: \"019c88cd-243c-4137-af28-55e47b730d1f\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh"
Jul 02 13:29:22.035544 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:22.035528    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/019c88cd-243c-4137-af28-55e47b730d1f-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh\" (UID: \"019c88cd-243c-4137-af28-55e47b730d1f\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh"
Jul 02 13:29:22.035622 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:22.035555    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/019c88cd-243c-4137-af28-55e47b730d1f-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh\" (UID: \"019c88cd-243c-4137-af28-55e47b730d1f\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh"
Jul 02 13:29:22.035622 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:22.035598    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/019c88cd-243c-4137-af28-55e47b730d1f-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh\" (UID: \"019c88cd-243c-4137-af28-55e47b730d1f\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh"
Jul 02 13:29:22.035690 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:22.035639    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/019c88cd-243c-4137-af28-55e47b730d1f-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh\" (UID: \"019c88cd-243c-4137-af28-55e47b730d1f\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh"
Jul 02 13:29:22.136455 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:22.136374    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/019c88cd-243c-4137-af28-55e47b730d1f-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh\" (UID: \"019c88cd-243c-4137-af28-55e47b730d1f\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh"
Jul 02 13:29:22.136646 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:22.136488    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/019c88cd-243c-4137-af28-55e47b730d1f-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh\" (UID: \"019c88cd-243c-4137-af28-55e47b730d1f\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh"
Jul 02 13:29:22.136646 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:22.136513    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/019c88cd-243c-4137-af28-55e47b730d1f-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh\" (UID: \"019c88cd-243c-4137-af28-55e47b730d1f\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh"
Jul 02 13:29:22.136646 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:22.136528    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/019c88cd-243c-4137-af28-55e47b730d1f-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh\" (UID: \"019c88cd-243c-4137-af28-55e47b730d1f\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh"
Jul 02 13:29:22.136646 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:22.136551    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/019c88cd-243c-4137-af28-55e47b730d1f-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh\" (UID: \"019c88cd-243c-4137-af28-55e47b730d1f\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh"
Jul 02 13:29:22.136646 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:22.136589    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vd7h\" (UniqueName: \"kubernetes.io/projected/019c88cd-243c-4137-af28-55e47b730d1f-kube-api-access-9vd7h\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh\" (UID: \"019c88cd-243c-4137-af28-55e47b730d1f\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh"
Jul 02 13:29:22.136981 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:22.136956    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/019c88cd-243c-4137-af28-55e47b730d1f-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh\" (UID: \"019c88cd-243c-4137-af28-55e47b730d1f\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh"
Jul 02 13:29:22.137051 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:22.137000    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/019c88cd-243c-4137-af28-55e47b730d1f-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh\" (UID: \"019c88cd-243c-4137-af28-55e47b730d1f\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh"
Jul 02 13:29:22.137102 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:22.137047    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/019c88cd-243c-4137-af28-55e47b730d1f-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh\" (UID: \"019c88cd-243c-4137-af28-55e47b730d1f\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh"
Jul 02 13:29:22.138868 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:22.138839    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/019c88cd-243c-4137-af28-55e47b730d1f-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh\" (UID: \"019c88cd-243c-4137-af28-55e47b730d1f\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh"
Jul 02 13:29:22.139050 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:22.139031    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/019c88cd-243c-4137-af28-55e47b730d1f-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh\" (UID: \"019c88cd-243c-4137-af28-55e47b730d1f\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh"
Jul 02 13:29:22.145926 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:22.145898    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vd7h\" (UniqueName: \"kubernetes.io/projected/019c88cd-243c-4137-af28-55e47b730d1f-kube-api-access-9vd7h\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh\" (UID: \"019c88cd-243c-4137-af28-55e47b730d1f\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh"
Jul 02 13:29:22.312192 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:22.312094    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh"
Jul 02 13:29:22.351042 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:22.351008    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:29:22.456499 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:22.456470    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh"]
Jul 02 13:29:22.459278 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:29:22.459244    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod019c88cd_243c_4137_af28_55e47b730d1f.slice/crio-94dc7ef4c640e7aaf0447802c2321545f440935329d114e237fa45d28a8b866a WatchSource:0}: Error finding container 94dc7ef4c640e7aaf0447802c2321545f440935329d114e237fa45d28a8b866a: Status 404 returned error can't find the container with id 94dc7ef4c640e7aaf0447802c2321545f440935329d114e237fa45d28a8b866a
Jul 02 13:29:22.646598 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:22.646560    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh" event={"ID":"019c88cd-243c-4137-af28-55e47b730d1f","Type":"ContainerStarted","Data":"25f8cb2a34be05a40cdc3b57ba6569a16f7cd5bad1bfb199a393283c10c02878"}
Jul 02 13:29:22.646786 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:22.646618    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh" event={"ID":"019c88cd-243c-4137-af28-55e47b730d1f","Type":"ContainerStarted","Data":"94dc7ef4c640e7aaf0447802c2321545f440935329d114e237fa45d28a8b866a"}
Jul 02 13:29:22.669095 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:22.669033    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh" podStartSLOduration=1.669016925 podStartE2EDuration="1.669016925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:29:22.668008345 +0000 UTC m=+873.895919519" watchObservedRunningTime="2026-07-02 13:29:22.669016925 +0000 UTC m=+873.896928098"
Jul 02 13:29:23.312880 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:23.312842    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh"
Jul 02 13:29:32.313134 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:32.313092    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh"
Jul 02 13:29:32.325603 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:32.325571    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh"
Jul 02 13:29:32.707523 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:32.707486    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh"
Jul 02 13:29:45.394028 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:45.393943    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:29:49.247934 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:49.247894    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 13:29:49.251142 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:29:49.251116    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 13:30:18.469019 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:30:18.468982    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:30:18.598628 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:30:18.598593    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:30:44.309111 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:30:44.309051    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:30:44.653284 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:30:44.653248    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:30:53.541307 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:30:53.541273    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:30:54.787635 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:30:54.787602    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:31:19.534873 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:19.534792    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:31:19.835774 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:19.835688    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb"]
Jul 02 13:31:19.838806 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:19.838790    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb"
Jul 02 13:31:19.843371 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:19.843348    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-nqjp2\""
Jul 02 13:31:19.843523 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:19.843411    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 02 13:31:19.854005 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:19.853973    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb"]
Jul 02 13:31:19.937156 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:19.937118    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cd0e902e-fbcd-4759-8823-64fe6cc2bdc8-tokenizer-uds\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb\" (UID: \"cd0e902e-fbcd-4759-8823-64fe6cc2bdc8\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb"
Jul 02 13:31:19.937327 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:19.937176    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd0e902e-fbcd-4759-8823-64fe6cc2bdc8-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb\" (UID: \"cd0e902e-fbcd-4759-8823-64fe6cc2bdc8\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb"
Jul 02 13:31:19.937327 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:19.937203    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmhml\" (UniqueName: \"kubernetes.io/projected/cd0e902e-fbcd-4759-8823-64fe6cc2bdc8-kube-api-access-zmhml\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb\" (UID: \"cd0e902e-fbcd-4759-8823-64fe6cc2bdc8\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb"
Jul 02 13:31:20.038505 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:20.038464    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cd0e902e-fbcd-4759-8823-64fe6cc2bdc8-tokenizer-uds\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb\" (UID: \"cd0e902e-fbcd-4759-8823-64fe6cc2bdc8\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb"
Jul 02 13:31:20.038694 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:20.038524    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd0e902e-fbcd-4759-8823-64fe6cc2bdc8-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb\" (UID: \"cd0e902e-fbcd-4759-8823-64fe6cc2bdc8\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb"
Jul 02 13:31:20.038694 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:20.038643    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmhml\" (UniqueName: \"kubernetes.io/projected/cd0e902e-fbcd-4759-8823-64fe6cc2bdc8-kube-api-access-zmhml\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb\" (UID: \"cd0e902e-fbcd-4759-8823-64fe6cc2bdc8\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb"
Jul 02 13:31:20.038910 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:20.038888    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cd0e902e-fbcd-4759-8823-64fe6cc2bdc8-tokenizer-uds\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb\" (UID: \"cd0e902e-fbcd-4759-8823-64fe6cc2bdc8\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb"
Jul 02 13:31:20.040988 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:20.040970    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd0e902e-fbcd-4759-8823-64fe6cc2bdc8-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb\" (UID: \"cd0e902e-fbcd-4759-8823-64fe6cc2bdc8\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb"
Jul 02 13:31:20.048948 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:20.048923    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmhml\" (UniqueName: \"kubernetes.io/projected/cd0e902e-fbcd-4759-8823-64fe6cc2bdc8-kube-api-access-zmhml\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb\" (UID: \"cd0e902e-fbcd-4759-8823-64fe6cc2bdc8\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb"
Jul 02 13:31:20.148335 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:20.148296    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb"
Jul 02 13:31:20.284261 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:20.284233    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb"]
Jul 02 13:31:20.286371 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:31:20.286337    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd0e902e_fbcd_4759_8823_64fe6cc2bdc8.slice/crio-62a047248aebd640df00d638ba662ddcc7bb74596a7fad1f58a735faf9c5b948 WatchSource:0}: Error finding container 62a047248aebd640df00d638ba662ddcc7bb74596a7fad1f58a735faf9c5b948: Status 404 returned error can't find the container with id 62a047248aebd640df00d638ba662ddcc7bb74596a7fad1f58a735faf9c5b948
Jul 02 13:31:21.247346 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:21.247308    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb" event={"ID":"cd0e902e-fbcd-4759-8823-64fe6cc2bdc8","Type":"ContainerStarted","Data":"3bf490efb25af20323eb8f0ab2522d9826bd04357123802cc1aebac612233cc3"}
Jul 02 13:31:21.247782 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:21.247363    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb" event={"ID":"cd0e902e-fbcd-4759-8823-64fe6cc2bdc8","Type":"ContainerStarted","Data":"62a047248aebd640df00d638ba662ddcc7bb74596a7fad1f58a735faf9c5b948"}
Jul 02 13:31:21.247782 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:21.247410    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb"
Jul 02 13:31:21.265873 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:21.265820    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb" podStartSLOduration=2.265798701 podStartE2EDuration="2.265798701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:31:21.264675588 +0000 UTC m=+992.492586762" watchObservedRunningTime="2026-07-02 13:31:21.265798701 +0000 UTC m=+992.493709882"
Jul 02 13:31:21.363590 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:21.363553    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:31:28.540017 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:28.539982    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:31:52.254016 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:52.253985    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb"
Jul 02 13:31:53.577770 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:53.577733    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb"]
Jul 02 13:31:53.588302 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:53.588267    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:31:53.590308 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:31:53.590267    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 02 13:31:53.590308 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:31:53.590308 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:31:53.590308 ip-10-0-135-143 kubenswrapper[2583]:  > containerID="3bf490efb25af20323eb8f0ab2522d9826bd04357123802cc1aebac612233cc3" cmd=["/bin/sleep","15"]
Jul 02 13:31:53.590625 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:53.590333    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb" message="" err=<
Jul 02 13:31:53.590625 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:31:53.590625 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:31:53.590625 ip-10-0-135-143 kubenswrapper[2583]:  >
Jul 02 13:31:53.590625 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:31:53.590353    2583 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 02 13:31:53.590625 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:31:53.590625 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:31:53.590625 ip-10-0-135-143 kubenswrapper[2583]:  > pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb" podUID="cd0e902e-fbcd-4759-8823-64fe6cc2bdc8" containerName="main" containerID="cri-o://3bf490efb25af20323eb8f0ab2522d9826bd04357123802cc1aebac612233cc3"
Jul 02 13:31:53.590625 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:53.590446    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb" podUID="cd0e902e-fbcd-4759-8823-64fe6cc2bdc8" containerName="main" containerID="cri-o://3bf490efb25af20323eb8f0ab2522d9826bd04357123802cc1aebac612233cc3" gracePeriod=60
Jul 02 13:31:53.674112 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:53.674084    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:31:53.845323 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:53.845293    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb"
Jul 02 13:31:53.949036 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:53.948996    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmhml\" (UniqueName: \"kubernetes.io/projected/cd0e902e-fbcd-4759-8823-64fe6cc2bdc8-kube-api-access-zmhml\") pod \"cd0e902e-fbcd-4759-8823-64fe6cc2bdc8\" (UID: \"cd0e902e-fbcd-4759-8823-64fe6cc2bdc8\") "
Jul 02 13:31:53.949231 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:53.949099    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cd0e902e-fbcd-4759-8823-64fe6cc2bdc8-tokenizer-uds\") pod \"cd0e902e-fbcd-4759-8823-64fe6cc2bdc8\" (UID: \"cd0e902e-fbcd-4759-8823-64fe6cc2bdc8\") "
Jul 02 13:31:53.949231 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:53.949157    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd0e902e-fbcd-4759-8823-64fe6cc2bdc8-tls-certs\") pod \"cd0e902e-fbcd-4759-8823-64fe6cc2bdc8\" (UID: \"cd0e902e-fbcd-4759-8823-64fe6cc2bdc8\") "
Jul 02 13:31:53.949378 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:53.949353    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd0e902e-fbcd-4759-8823-64fe6cc2bdc8-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "cd0e902e-fbcd-4759-8823-64fe6cc2bdc8" (UID: "cd0e902e-fbcd-4759-8823-64fe6cc2bdc8"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:31:53.949565 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:53.949542    2583 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cd0e902e-fbcd-4759-8823-64fe6cc2bdc8-tokenizer-uds\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:31:53.951172 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:53.951151    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd0e902e-fbcd-4759-8823-64fe6cc2bdc8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd0e902e-fbcd-4759-8823-64fe6cc2bdc8" (UID: "cd0e902e-fbcd-4759-8823-64fe6cc2bdc8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:31:53.951297 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:53.951276    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd0e902e-fbcd-4759-8823-64fe6cc2bdc8-kube-api-access-zmhml" (OuterVolumeSpecName: "kube-api-access-zmhml") pod "cd0e902e-fbcd-4759-8823-64fe6cc2bdc8" (UID: "cd0e902e-fbcd-4759-8823-64fe6cc2bdc8"). InnerVolumeSpecName "kube-api-access-zmhml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:31:54.056577 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:54.051129    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd0e902e-fbcd-4759-8823-64fe6cc2bdc8-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:31:54.056577 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:54.051168    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmhml\" (UniqueName: \"kubernetes.io/projected/cd0e902e-fbcd-4759-8823-64fe6cc2bdc8-kube-api-access-zmhml\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:31:54.413454 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:54.413413    2583 generic.go:356] "Generic (PLEG): container finished" podID="cd0e902e-fbcd-4759-8823-64fe6cc2bdc8" containerID="3bf490efb25af20323eb8f0ab2522d9826bd04357123802cc1aebac612233cc3" exitCode=0
Jul 02 13:31:54.413653 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:54.413474    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb"
Jul 02 13:31:54.413653 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:54.413489    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb" event={"ID":"cd0e902e-fbcd-4759-8823-64fe6cc2bdc8","Type":"ContainerDied","Data":"3bf490efb25af20323eb8f0ab2522d9826bd04357123802cc1aebac612233cc3"}
Jul 02 13:31:54.413653 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:54.413550    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb" event={"ID":"cd0e902e-fbcd-4759-8823-64fe6cc2bdc8","Type":"ContainerDied","Data":"62a047248aebd640df00d638ba662ddcc7bb74596a7fad1f58a735faf9c5b948"}
Jul 02 13:31:54.413653 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:54.413571    2583 scope.go:117] "RemoveContainer" containerID="3bf490efb25af20323eb8f0ab2522d9826bd04357123802cc1aebac612233cc3"
Jul 02 13:31:54.422574 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:54.422554    2583 scope.go:117] "RemoveContainer" containerID="3bf490efb25af20323eb8f0ab2522d9826bd04357123802cc1aebac612233cc3"
Jul 02 13:31:54.422856 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:31:54.422832    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bf490efb25af20323eb8f0ab2522d9826bd04357123802cc1aebac612233cc3\": container with ID starting with 3bf490efb25af20323eb8f0ab2522d9826bd04357123802cc1aebac612233cc3 not found: ID does not exist" containerID="3bf490efb25af20323eb8f0ab2522d9826bd04357123802cc1aebac612233cc3"
Jul 02 13:31:54.422927 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:54.422862    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bf490efb25af20323eb8f0ab2522d9826bd04357123802cc1aebac612233cc3"} err="failed to get container status \"3bf490efb25af20323eb8f0ab2522d9826bd04357123802cc1aebac612233cc3\": rpc error: code = NotFound desc = could not find container \"3bf490efb25af20323eb8f0ab2522d9826bd04357123802cc1aebac612233cc3\": container with ID starting with 3bf490efb25af20323eb8f0ab2522d9826bd04357123802cc1aebac612233cc3 not found: ID does not exist"
Jul 02 13:31:54.438296 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:54.438263    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb"]
Jul 02 13:31:54.439954 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:54.439935    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schef96bb"]
Jul 02 13:31:55.233947 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:31:55.233904    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd0e902e-fbcd-4759-8823-64fe6cc2bdc8" path="/var/lib/kubelet/pods/cd0e902e-fbcd-4759-8823-64fe6cc2bdc8/volumes"
Jul 02 13:32:13.514672 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:13.514632    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:32:13.515430 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:13.515398    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh"]
Jul 02 13:32:13.598369 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:13.598335    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:32:28.536411 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:28.536337    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh" podUID="019c88cd-243c-4137-af28-55e47b730d1f" containerName="main" containerID="cri-o://25f8cb2a34be05a40cdc3b57ba6569a16f7cd5bad1bfb199a393283c10c02878" gracePeriod=45
Jul 02 13:32:28.696542 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:28.696488    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh" podUID="019c88cd-243c-4137-af28-55e47b730d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 02 13:32:28.795268 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:28.795200    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh"
Jul 02 13:32:28.873991 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:28.873952    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/019c88cd-243c-4137-af28-55e47b730d1f-tmp-dir\") pod \"019c88cd-243c-4137-af28-55e47b730d1f\" (UID: \"019c88cd-243c-4137-af28-55e47b730d1f\") "
Jul 02 13:32:28.874184 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:28.874032    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/019c88cd-243c-4137-af28-55e47b730d1f-model-cache\") pod \"019c88cd-243c-4137-af28-55e47b730d1f\" (UID: \"019c88cd-243c-4137-af28-55e47b730d1f\") "
Jul 02 13:32:28.874184 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:28.874056    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/019c88cd-243c-4137-af28-55e47b730d1f-tls-certs\") pod \"019c88cd-243c-4137-af28-55e47b730d1f\" (UID: \"019c88cd-243c-4137-af28-55e47b730d1f\") "
Jul 02 13:32:28.874273 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:28.874196    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/019c88cd-243c-4137-af28-55e47b730d1f-home\") pod \"019c88cd-243c-4137-af28-55e47b730d1f\" (UID: \"019c88cd-243c-4137-af28-55e47b730d1f\") "
Jul 02 13:32:28.874273 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:28.874244    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vd7h\" (UniqueName: \"kubernetes.io/projected/019c88cd-243c-4137-af28-55e47b730d1f-kube-api-access-9vd7h\") pod \"019c88cd-243c-4137-af28-55e47b730d1f\" (UID: \"019c88cd-243c-4137-af28-55e47b730d1f\") "
Jul 02 13:32:28.874419 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:28.874275    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/019c88cd-243c-4137-af28-55e47b730d1f-dshm\") pod \"019c88cd-243c-4137-af28-55e47b730d1f\" (UID: \"019c88cd-243c-4137-af28-55e47b730d1f\") "
Jul 02 13:32:28.874419 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:28.874289    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/019c88cd-243c-4137-af28-55e47b730d1f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "019c88cd-243c-4137-af28-55e47b730d1f" (UID: "019c88cd-243c-4137-af28-55e47b730d1f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:32:28.874419 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:28.874306    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/019c88cd-243c-4137-af28-55e47b730d1f-model-cache" (OuterVolumeSpecName: "model-cache") pod "019c88cd-243c-4137-af28-55e47b730d1f" (UID: "019c88cd-243c-4137-af28-55e47b730d1f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:32:28.874580 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:28.874471    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/019c88cd-243c-4137-af28-55e47b730d1f-home" (OuterVolumeSpecName: "home") pod "019c88cd-243c-4137-af28-55e47b730d1f" (UID: "019c88cd-243c-4137-af28-55e47b730d1f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:32:28.874662 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:28.874643    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/019c88cd-243c-4137-af28-55e47b730d1f-home\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:32:28.874718 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:28.874668    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/019c88cd-243c-4137-af28-55e47b730d1f-tmp-dir\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:32:28.874718 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:28.874686    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/019c88cd-243c-4137-af28-55e47b730d1f-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:32:28.876320 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:28.876297    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/019c88cd-243c-4137-af28-55e47b730d1f-dshm" (OuterVolumeSpecName: "dshm") pod "019c88cd-243c-4137-af28-55e47b730d1f" (UID: "019c88cd-243c-4137-af28-55e47b730d1f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:32:28.876419 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:28.876298    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/019c88cd-243c-4137-af28-55e47b730d1f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "019c88cd-243c-4137-af28-55e47b730d1f" (UID: "019c88cd-243c-4137-af28-55e47b730d1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:32:28.876419 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:28.876319    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/019c88cd-243c-4137-af28-55e47b730d1f-kube-api-access-9vd7h" (OuterVolumeSpecName: "kube-api-access-9vd7h") pod "019c88cd-243c-4137-af28-55e47b730d1f" (UID: "019c88cd-243c-4137-af28-55e47b730d1f"). InnerVolumeSpecName "kube-api-access-9vd7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:32:28.975639 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:28.975586    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vd7h\" (UniqueName: \"kubernetes.io/projected/019c88cd-243c-4137-af28-55e47b730d1f-kube-api-access-9vd7h\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:32:28.975639 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:28.975635    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/019c88cd-243c-4137-af28-55e47b730d1f-dshm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:32:28.975639 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:28.975647    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/019c88cd-243c-4137-af28-55e47b730d1f-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:32:29.594881 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:29.594796    2583 generic.go:356] "Generic (PLEG): container finished" podID="019c88cd-243c-4137-af28-55e47b730d1f" containerID="25f8cb2a34be05a40cdc3b57ba6569a16f7cd5bad1bfb199a393283c10c02878" exitCode=0
Jul 02 13:32:29.594881 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:29.594852    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh"
Jul 02 13:32:29.595304 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:29.594881    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh" event={"ID":"019c88cd-243c-4137-af28-55e47b730d1f","Type":"ContainerDied","Data":"25f8cb2a34be05a40cdc3b57ba6569a16f7cd5bad1bfb199a393283c10c02878"}
Jul 02 13:32:29.595304 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:29.594932    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh" event={"ID":"019c88cd-243c-4137-af28-55e47b730d1f","Type":"ContainerDied","Data":"94dc7ef4c640e7aaf0447802c2321545f440935329d114e237fa45d28a8b866a"}
Jul 02 13:32:29.595304 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:29.594955    2583 scope.go:117] "RemoveContainer" containerID="25f8cb2a34be05a40cdc3b57ba6569a16f7cd5bad1bfb199a393283c10c02878"
Jul 02 13:32:29.603641 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:29.603620    2583 scope.go:117] "RemoveContainer" containerID="25f8cb2a34be05a40cdc3b57ba6569a16f7cd5bad1bfb199a393283c10c02878"
Jul 02 13:32:29.603937 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:32:29.603920    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25f8cb2a34be05a40cdc3b57ba6569a16f7cd5bad1bfb199a393283c10c02878\": container with ID starting with 25f8cb2a34be05a40cdc3b57ba6569a16f7cd5bad1bfb199a393283c10c02878 not found: ID does not exist" containerID="25f8cb2a34be05a40cdc3b57ba6569a16f7cd5bad1bfb199a393283c10c02878"
Jul 02 13:32:29.604000 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:29.603949    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f8cb2a34be05a40cdc3b57ba6569a16f7cd5bad1bfb199a393283c10c02878"} err="failed to get container status \"25f8cb2a34be05a40cdc3b57ba6569a16f7cd5bad1bfb199a393283c10c02878\": rpc error: code = NotFound desc = could not find container \"25f8cb2a34be05a40cdc3b57ba6569a16f7cd5bad1bfb199a393283c10c02878\": container with ID starting with 25f8cb2a34be05a40cdc3b57ba6569a16f7cd5bad1bfb199a393283c10c02878 not found: ID does not exist"
Jul 02 13:32:29.614890 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:29.614857    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh"]
Jul 02 13:32:29.618883 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:29.618859    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4grrqh"]
Jul 02 13:32:31.233755 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:31.233717    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="019c88cd-243c-4137-af28-55e47b730d1f" path="/var/lib/kubelet/pods/019c88cd-243c-4137-af28-55e47b730d1f/volumes"
Jul 02 13:32:43.199201 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.199120    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk"]
Jul 02 13:32:43.199596 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.199505    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd0e902e-fbcd-4759-8823-64fe6cc2bdc8" containerName="main"
Jul 02 13:32:43.199596 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.199517    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd0e902e-fbcd-4759-8823-64fe6cc2bdc8" containerName="main"
Jul 02 13:32:43.199596 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.199534    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="019c88cd-243c-4137-af28-55e47b730d1f" containerName="main"
Jul 02 13:32:43.199596 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.199539    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="019c88cd-243c-4137-af28-55e47b730d1f" containerName="main"
Jul 02 13:32:43.199596 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.199597    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd0e902e-fbcd-4759-8823-64fe6cc2bdc8" containerName="main"
Jul 02 13:32:43.199783 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.199604    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="019c88cd-243c-4137-af28-55e47b730d1f" containerName="main"
Jul 02 13:32:43.203850 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.203826    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk"
Jul 02 13:32:43.208141 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.208117    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7s8sh\""
Jul 02 13:32:43.208257 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.208158    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Jul 02 13:32:43.217926 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.217895    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk"]
Jul 02 13:32:43.234820 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.234784    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:32:43.302834 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.302786    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fd91d44-aa44-4a5d-9277-262ca1224372-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk\" (UID: \"9fd91d44-aa44-4a5d-9277-262ca1224372\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk"
Jul 02 13:32:43.302834 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.302836    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fd91d44-aa44-4a5d-9277-262ca1224372-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk\" (UID: \"9fd91d44-aa44-4a5d-9277-262ca1224372\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk"
Jul 02 13:32:43.303114 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.302876    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fd91d44-aa44-4a5d-9277-262ca1224372-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk\" (UID: \"9fd91d44-aa44-4a5d-9277-262ca1224372\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk"
Jul 02 13:32:43.303114 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.302986    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fd91d44-aa44-4a5d-9277-262ca1224372-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk\" (UID: \"9fd91d44-aa44-4a5d-9277-262ca1224372\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk"
Jul 02 13:32:43.303237 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.303120    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fd91d44-aa44-4a5d-9277-262ca1224372-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk\" (UID: \"9fd91d44-aa44-4a5d-9277-262ca1224372\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk"
Jul 02 13:32:43.303237 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.303209    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t97x\" (UniqueName: \"kubernetes.io/projected/9fd91d44-aa44-4a5d-9277-262ca1224372-kube-api-access-5t97x\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk\" (UID: \"9fd91d44-aa44-4a5d-9277-262ca1224372\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk"
Jul 02 13:32:43.404042 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.404000    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fd91d44-aa44-4a5d-9277-262ca1224372-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk\" (UID: \"9fd91d44-aa44-4a5d-9277-262ca1224372\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk"
Jul 02 13:32:43.404245 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.404066    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fd91d44-aa44-4a5d-9277-262ca1224372-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk\" (UID: \"9fd91d44-aa44-4a5d-9277-262ca1224372\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk"
Jul 02 13:32:43.404245 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.404103    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t97x\" (UniqueName: \"kubernetes.io/projected/9fd91d44-aa44-4a5d-9277-262ca1224372-kube-api-access-5t97x\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk\" (UID: \"9fd91d44-aa44-4a5d-9277-262ca1224372\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk"
Jul 02 13:32:43.404245 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.404157    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fd91d44-aa44-4a5d-9277-262ca1224372-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk\" (UID: \"9fd91d44-aa44-4a5d-9277-262ca1224372\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk"
Jul 02 13:32:43.404245 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.404182    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fd91d44-aa44-4a5d-9277-262ca1224372-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk\" (UID: \"9fd91d44-aa44-4a5d-9277-262ca1224372\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk"
Jul 02 13:32:43.404245 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.404203    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fd91d44-aa44-4a5d-9277-262ca1224372-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk\" (UID: \"9fd91d44-aa44-4a5d-9277-262ca1224372\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk"
Jul 02 13:32:43.404587 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.404563    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fd91d44-aa44-4a5d-9277-262ca1224372-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk\" (UID: \"9fd91d44-aa44-4a5d-9277-262ca1224372\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk"
Jul 02 13:32:43.404648 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.404601    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fd91d44-aa44-4a5d-9277-262ca1224372-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk\" (UID: \"9fd91d44-aa44-4a5d-9277-262ca1224372\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk"
Jul 02 13:32:43.404695 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.404646    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fd91d44-aa44-4a5d-9277-262ca1224372-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk\" (UID: \"9fd91d44-aa44-4a5d-9277-262ca1224372\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk"
Jul 02 13:32:43.406293 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.406269    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fd91d44-aa44-4a5d-9277-262ca1224372-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk\" (UID: \"9fd91d44-aa44-4a5d-9277-262ca1224372\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk"
Jul 02 13:32:43.406666 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.406647    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fd91d44-aa44-4a5d-9277-262ca1224372-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk\" (UID: \"9fd91d44-aa44-4a5d-9277-262ca1224372\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk"
Jul 02 13:32:43.414181 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.414152    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t97x\" (UniqueName: \"kubernetes.io/projected/9fd91d44-aa44-4a5d-9277-262ca1224372-kube-api-access-5t97x\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk\" (UID: \"9fd91d44-aa44-4a5d-9277-262ca1224372\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk"
Jul 02 13:32:43.513505 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.513414    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk"
Jul 02 13:32:43.564045 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.564011    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:32:43.654370 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.654342    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk"]
Jul 02 13:32:43.657361 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:32:43.657333    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fd91d44_aa44_4a5d_9277_262ca1224372.slice/crio-2825bfc21128ec03b31d52d6273ee4ab92857e8908ae70c5701f3e1db2f96ebf WatchSource:0}: Error finding container 2825bfc21128ec03b31d52d6273ee4ab92857e8908ae70c5701f3e1db2f96ebf: Status 404 returned error can't find the container with id 2825bfc21128ec03b31d52d6273ee4ab92857e8908ae70c5701f3e1db2f96ebf
Jul 02 13:32:43.668085 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:43.668053    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk" event={"ID":"9fd91d44-aa44-4a5d-9277-262ca1224372","Type":"ContainerStarted","Data":"2825bfc21128ec03b31d52d6273ee4ab92857e8908ae70c5701f3e1db2f96ebf"}
Jul 02 13:32:44.674293 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:44.674249    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk" event={"ID":"9fd91d44-aa44-4a5d-9277-262ca1224372","Type":"ContainerStarted","Data":"dbd820eca4751551dfb228fd7a960d1226759a1e733cd5108142b0fe11db97d5"}
Jul 02 13:32:44.696256 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:44.696191    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk" podStartSLOduration=1.696173597 podStartE2EDuration="1.696173597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:32:44.694909021 +0000 UTC m=+1075.922820194" watchObservedRunningTime="2026-07-02 13:32:44.696173597 +0000 UTC m=+1075.924084770"
Jul 02 13:32:45.514447 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:45.514383    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk"
Jul 02 13:32:53.454596 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:53.454557    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:32:53.513841 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:53.513805    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk"
Jul 02 13:32:53.528077 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:53.528048    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk"
Jul 02 13:32:53.727756 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:53.727672    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk"
Jul 02 13:32:54.915673 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:54.915623    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:32:54.981318 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:32:54.981280    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:33:36.891617 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:33:36.891579    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:33:37.005658 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:33:37.005618    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:33:47.392098 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:33:47.392061    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:33:47.597809 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:33:47.597761    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:33:59.823629 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:33:59.823588    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:34:18.409314 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:18.409225    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:34:18.636990 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:18.636951    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff"]
Jul 02 13:34:18.640869 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:18.640844    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff"
Jul 02 13:34:18.644915 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:18.644884    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-scgcn\""
Jul 02 13:34:18.645042 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:18.644922    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 02 13:34:18.654112 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:18.654079    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff"]
Jul 02 13:34:18.668856 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:18.668782    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:34:18.772375 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:18.772332    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnf9w\" (UniqueName: \"kubernetes.io/projected/67907c8f-1d3f-4959-a69f-65f0eeee9693-kube-api-access-xnf9w\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff\" (UID: \"67907c8f-1d3f-4959-a69f-65f0eeee9693\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff"
Jul 02 13:34:18.772570 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:18.772380    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/67907c8f-1d3f-4959-a69f-65f0eeee9693-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff\" (UID: \"67907c8f-1d3f-4959-a69f-65f0eeee9693\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff"
Jul 02 13:34:18.772570 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:18.772437    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67907c8f-1d3f-4959-a69f-65f0eeee9693-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff\" (UID: \"67907c8f-1d3f-4959-a69f-65f0eeee9693\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff"
Jul 02 13:34:18.873117 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:18.873078    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnf9w\" (UniqueName: \"kubernetes.io/projected/67907c8f-1d3f-4959-a69f-65f0eeee9693-kube-api-access-xnf9w\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff\" (UID: \"67907c8f-1d3f-4959-a69f-65f0eeee9693\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff"
Jul 02 13:34:18.873307 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:18.873128    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/67907c8f-1d3f-4959-a69f-65f0eeee9693-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff\" (UID: \"67907c8f-1d3f-4959-a69f-65f0eeee9693\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff"
Jul 02 13:34:18.873307 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:18.873170    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67907c8f-1d3f-4959-a69f-65f0eeee9693-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff\" (UID: \"67907c8f-1d3f-4959-a69f-65f0eeee9693\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff"
Jul 02 13:34:18.873577 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:18.873556    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/67907c8f-1d3f-4959-a69f-65f0eeee9693-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff\" (UID: \"67907c8f-1d3f-4959-a69f-65f0eeee9693\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff"
Jul 02 13:34:18.875745 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:18.875725    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67907c8f-1d3f-4959-a69f-65f0eeee9693-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff\" (UID: \"67907c8f-1d3f-4959-a69f-65f0eeee9693\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff"
Jul 02 13:34:18.881608 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:18.881581    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnf9w\" (UniqueName: \"kubernetes.io/projected/67907c8f-1d3f-4959-a69f-65f0eeee9693-kube-api-access-xnf9w\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff\" (UID: \"67907c8f-1d3f-4959-a69f-65f0eeee9693\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff"
Jul 02 13:34:18.950209 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:18.950111    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff"
Jul 02 13:34:19.095254 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:19.095228    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff"]
Jul 02 13:34:19.097882 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:34:19.097848    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67907c8f_1d3f_4959_a69f_65f0eeee9693.slice/crio-4e255ec9bcbd1c20f007991b22761adb1cfd7c40f7715eba2ac5ec6c25911ad4 WatchSource:0}: Error finding container 4e255ec9bcbd1c20f007991b22761adb1cfd7c40f7715eba2ac5ec6c25911ad4: Status 404 returned error can't find the container with id 4e255ec9bcbd1c20f007991b22761adb1cfd7c40f7715eba2ac5ec6c25911ad4
Jul 02 13:34:19.138051 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:19.138016    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff" event={"ID":"67907c8f-1d3f-4959-a69f-65f0eeee9693","Type":"ContainerStarted","Data":"4e255ec9bcbd1c20f007991b22761adb1cfd7c40f7715eba2ac5ec6c25911ad4"}
Jul 02 13:34:20.144812 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:20.144778    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff" event={"ID":"67907c8f-1d3f-4959-a69f-65f0eeee9693","Type":"ContainerStarted","Data":"bb0e0f5643c3da077525d85b1717958062c5de468df692f5d146ab87d9575641"}
Jul 02 13:34:20.145209 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:20.145036    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff"
Jul 02 13:34:20.164107 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:20.164051    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff" podStartSLOduration=2.164033815 podStartE2EDuration="2.164033815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:34:20.161905552 +0000 UTC m=+1171.389816726" watchObservedRunningTime="2026-07-02 13:34:20.164033815 +0000 UTC m=+1171.391945020"
Jul 02 13:34:28.439044 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:28.439009    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:34:49.280011 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:49.279983    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 13:34:49.283964 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:49.283933    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 13:34:51.151704 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:51.151670    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff"
Jul 02 13:34:52.248790 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:52.248756    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:34:52.259414 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:52.259350    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff"]
Jul 02 13:34:52.269407 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:34:52.269337    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 02 13:34:52.269407 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:34:52.269407 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:34:52.269407 ip-10-0-135-143 kubenswrapper[2583]:  > containerID="bb0e0f5643c3da077525d85b1717958062c5de468df692f5d146ab87d9575641" cmd=["/bin/sleep","15"]
Jul 02 13:34:52.270228 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:52.269431    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff" message="" err=<
Jul 02 13:34:52.270228 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:34:52.270228 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:34:52.270228 ip-10-0-135-143 kubenswrapper[2583]:  >
Jul 02 13:34:52.270228 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:34:52.269450    2583 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 02 13:34:52.270228 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:34:52.270228 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:34:52.270228 ip-10-0-135-143 kubenswrapper[2583]:  > pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff" podUID="67907c8f-1d3f-4959-a69f-65f0eeee9693" containerName="main" containerID="cri-o://bb0e0f5643c3da077525d85b1717958062c5de468df692f5d146ab87d9575641"
Jul 02 13:34:52.270228 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:52.269512    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff" podUID="67907c8f-1d3f-4959-a69f-65f0eeee9693" containerName="main" containerID="cri-o://bb0e0f5643c3da077525d85b1717958062c5de468df692f5d146ab87d9575641" gracePeriod=60
Jul 02 13:34:52.320908 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:52.320877    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:34:52.523751 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:52.523717    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff"
Jul 02 13:34:52.589431 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:52.589367    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67907c8f-1d3f-4959-a69f-65f0eeee9693-tls-certs\") pod \"67907c8f-1d3f-4959-a69f-65f0eeee9693\" (UID: \"67907c8f-1d3f-4959-a69f-65f0eeee9693\") "
Jul 02 13:34:52.589636 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:52.589459    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnf9w\" (UniqueName: \"kubernetes.io/projected/67907c8f-1d3f-4959-a69f-65f0eeee9693-kube-api-access-xnf9w\") pod \"67907c8f-1d3f-4959-a69f-65f0eeee9693\" (UID: \"67907c8f-1d3f-4959-a69f-65f0eeee9693\") "
Jul 02 13:34:52.589636 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:52.589592    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/67907c8f-1d3f-4959-a69f-65f0eeee9693-tokenizer-uds\") pod \"67907c8f-1d3f-4959-a69f-65f0eeee9693\" (UID: \"67907c8f-1d3f-4959-a69f-65f0eeee9693\") "
Jul 02 13:34:52.589903 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:52.589861    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67907c8f-1d3f-4959-a69f-65f0eeee9693-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "67907c8f-1d3f-4959-a69f-65f0eeee9693" (UID: "67907c8f-1d3f-4959-a69f-65f0eeee9693"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:34:52.591670 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:52.591641    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67907c8f-1d3f-4959-a69f-65f0eeee9693-kube-api-access-xnf9w" (OuterVolumeSpecName: "kube-api-access-xnf9w") pod "67907c8f-1d3f-4959-a69f-65f0eeee9693" (UID: "67907c8f-1d3f-4959-a69f-65f0eeee9693"). InnerVolumeSpecName "kube-api-access-xnf9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:34:52.591779 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:52.591718    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67907c8f-1d3f-4959-a69f-65f0eeee9693-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67907c8f-1d3f-4959-a69f-65f0eeee9693" (UID: "67907c8f-1d3f-4959-a69f-65f0eeee9693"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:34:52.690879 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:52.690825    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67907c8f-1d3f-4959-a69f-65f0eeee9693-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:34:52.690879 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:52.690875    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnf9w\" (UniqueName: \"kubernetes.io/projected/67907c8f-1d3f-4959-a69f-65f0eeee9693-kube-api-access-xnf9w\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:34:52.690879 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:52.690890    2583 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/67907c8f-1d3f-4959-a69f-65f0eeee9693-tokenizer-uds\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:34:53.306718 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:53.306681    2583 generic.go:356] "Generic (PLEG): container finished" podID="67907c8f-1d3f-4959-a69f-65f0eeee9693" containerID="bb0e0f5643c3da077525d85b1717958062c5de468df692f5d146ab87d9575641" exitCode=0
Jul 02 13:34:53.307176 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:53.306723    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff" event={"ID":"67907c8f-1d3f-4959-a69f-65f0eeee9693","Type":"ContainerDied","Data":"bb0e0f5643c3da077525d85b1717958062c5de468df692f5d146ab87d9575641"}
Jul 02 13:34:53.307176 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:53.306766    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff" event={"ID":"67907c8f-1d3f-4959-a69f-65f0eeee9693","Type":"ContainerDied","Data":"4e255ec9bcbd1c20f007991b22761adb1cfd7c40f7715eba2ac5ec6c25911ad4"}
Jul 02 13:34:53.307176 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:53.306791    2583 scope.go:117] "RemoveContainer" containerID="bb0e0f5643c3da077525d85b1717958062c5de468df692f5d146ab87d9575641"
Jul 02 13:34:53.307176 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:53.306724    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff"
Jul 02 13:34:53.316037 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:53.316007    2583 scope.go:117] "RemoveContainer" containerID="bb0e0f5643c3da077525d85b1717958062c5de468df692f5d146ab87d9575641"
Jul 02 13:34:53.316365 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:34:53.316335    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb0e0f5643c3da077525d85b1717958062c5de468df692f5d146ab87d9575641\": container with ID starting with bb0e0f5643c3da077525d85b1717958062c5de468df692f5d146ab87d9575641 not found: ID does not exist" containerID="bb0e0f5643c3da077525d85b1717958062c5de468df692f5d146ab87d9575641"
Jul 02 13:34:53.316449 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:53.316372    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb0e0f5643c3da077525d85b1717958062c5de468df692f5d146ab87d9575641"} err="failed to get container status \"bb0e0f5643c3da077525d85b1717958062c5de468df692f5d146ab87d9575641\": rpc error: code = NotFound desc = could not find container \"bb0e0f5643c3da077525d85b1717958062c5de468df692f5d146ab87d9575641\": container with ID starting with bb0e0f5643c3da077525d85b1717958062c5de468df692f5d146ab87d9575641 not found: ID does not exist"
Jul 02 13:34:53.325842 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:53.325807    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff"]
Jul 02 13:34:53.329765 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:53.329736    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqzzff"]
Jul 02 13:34:55.233748 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:34:55.233715    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67907c8f-1d3f-4959-a69f-65f0eeee9693" path="/var/lib/kubelet/pods/67907c8f-1d3f-4959-a69f-65f0eeee9693/volumes"
Jul 02 13:35:12.474925 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:12.474888    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk"]
Jul 02 13:35:12.494040 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:12.494007    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:35:12.554819 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:12.554769    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:35:27.495748 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:27.495691    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk" podUID="9fd91d44-aa44-4a5d-9277-262ca1224372" containerName="main" containerID="cri-o://dbd820eca4751551dfb228fd7a960d1226759a1e733cd5108142b0fe11db97d5" gracePeriod=45
Jul 02 13:35:27.749685 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:27.749615    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk"
Jul 02 13:35:27.813545 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:27.813509    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fd91d44-aa44-4a5d-9277-262ca1224372-tls-certs\") pod \"9fd91d44-aa44-4a5d-9277-262ca1224372\" (UID: \"9fd91d44-aa44-4a5d-9277-262ca1224372\") "
Jul 02 13:35:27.813713 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:27.813557    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fd91d44-aa44-4a5d-9277-262ca1224372-model-cache\") pod \"9fd91d44-aa44-4a5d-9277-262ca1224372\" (UID: \"9fd91d44-aa44-4a5d-9277-262ca1224372\") "
Jul 02 13:35:27.813713 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:27.813576    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5t97x\" (UniqueName: \"kubernetes.io/projected/9fd91d44-aa44-4a5d-9277-262ca1224372-kube-api-access-5t97x\") pod \"9fd91d44-aa44-4a5d-9277-262ca1224372\" (UID: \"9fd91d44-aa44-4a5d-9277-262ca1224372\") "
Jul 02 13:35:27.813713 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:27.813607    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fd91d44-aa44-4a5d-9277-262ca1224372-home\") pod \"9fd91d44-aa44-4a5d-9277-262ca1224372\" (UID: \"9fd91d44-aa44-4a5d-9277-262ca1224372\") "
Jul 02 13:35:27.813713 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:27.813629    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fd91d44-aa44-4a5d-9277-262ca1224372-dshm\") pod \"9fd91d44-aa44-4a5d-9277-262ca1224372\" (UID: \"9fd91d44-aa44-4a5d-9277-262ca1224372\") "
Jul 02 13:35:27.813713 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:27.813660    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fd91d44-aa44-4a5d-9277-262ca1224372-tmp-dir\") pod \"9fd91d44-aa44-4a5d-9277-262ca1224372\" (UID: \"9fd91d44-aa44-4a5d-9277-262ca1224372\") "
Jul 02 13:35:27.813986 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:27.813814    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd91d44-aa44-4a5d-9277-262ca1224372-model-cache" (OuterVolumeSpecName: "model-cache") pod "9fd91d44-aa44-4a5d-9277-262ca1224372" (UID: "9fd91d44-aa44-4a5d-9277-262ca1224372"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:35:27.813986 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:27.813903    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd91d44-aa44-4a5d-9277-262ca1224372-home" (OuterVolumeSpecName: "home") pod "9fd91d44-aa44-4a5d-9277-262ca1224372" (UID: "9fd91d44-aa44-4a5d-9277-262ca1224372"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:35:27.814072 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:27.813976    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd91d44-aa44-4a5d-9277-262ca1224372-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9fd91d44-aa44-4a5d-9277-262ca1224372" (UID: "9fd91d44-aa44-4a5d-9277-262ca1224372"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:35:27.814072 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:27.814067    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fd91d44-aa44-4a5d-9277-262ca1224372-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:35:27.814137 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:27.814082    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fd91d44-aa44-4a5d-9277-262ca1224372-home\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:35:27.814137 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:27.814095    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fd91d44-aa44-4a5d-9277-262ca1224372-tmp-dir\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:35:27.815706 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:27.815679    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fd91d44-aa44-4a5d-9277-262ca1224372-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9fd91d44-aa44-4a5d-9277-262ca1224372" (UID: "9fd91d44-aa44-4a5d-9277-262ca1224372"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:35:27.815828 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:27.815703    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd91d44-aa44-4a5d-9277-262ca1224372-dshm" (OuterVolumeSpecName: "dshm") pod "9fd91d44-aa44-4a5d-9277-262ca1224372" (UID: "9fd91d44-aa44-4a5d-9277-262ca1224372"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:35:27.815828 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:27.815796    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fd91d44-aa44-4a5d-9277-262ca1224372-kube-api-access-5t97x" (OuterVolumeSpecName: "kube-api-access-5t97x") pod "9fd91d44-aa44-4a5d-9277-262ca1224372" (UID: "9fd91d44-aa44-4a5d-9277-262ca1224372"). InnerVolumeSpecName "kube-api-access-5t97x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:35:27.915182 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:27.915143    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fd91d44-aa44-4a5d-9277-262ca1224372-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:35:27.915182 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:27.915174    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t97x\" (UniqueName: \"kubernetes.io/projected/9fd91d44-aa44-4a5d-9277-262ca1224372-kube-api-access-5t97x\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:35:27.915182 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:27.915184    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fd91d44-aa44-4a5d-9277-262ca1224372-dshm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:35:28.481324 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:28.481290    2583 generic.go:356] "Generic (PLEG): container finished" podID="9fd91d44-aa44-4a5d-9277-262ca1224372" containerID="dbd820eca4751551dfb228fd7a960d1226759a1e733cd5108142b0fe11db97d5" exitCode=0
Jul 02 13:35:28.481536 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:28.481335    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk"
Jul 02 13:35:28.481536 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:28.481374    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk" event={"ID":"9fd91d44-aa44-4a5d-9277-262ca1224372","Type":"ContainerDied","Data":"dbd820eca4751551dfb228fd7a960d1226759a1e733cd5108142b0fe11db97d5"}
Jul 02 13:35:28.481536 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:28.481453    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk" event={"ID":"9fd91d44-aa44-4a5d-9277-262ca1224372","Type":"ContainerDied","Data":"2825bfc21128ec03b31d52d6273ee4ab92857e8908ae70c5701f3e1db2f96ebf"}
Jul 02 13:35:28.481536 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:28.481477    2583 scope.go:117] "RemoveContainer" containerID="dbd820eca4751551dfb228fd7a960d1226759a1e733cd5108142b0fe11db97d5"
Jul 02 13:35:28.490554 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:28.490534    2583 scope.go:117] "RemoveContainer" containerID="dbd820eca4751551dfb228fd7a960d1226759a1e733cd5108142b0fe11db97d5"
Jul 02 13:35:28.490865 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:35:28.490845    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbd820eca4751551dfb228fd7a960d1226759a1e733cd5108142b0fe11db97d5\": container with ID starting with dbd820eca4751551dfb228fd7a960d1226759a1e733cd5108142b0fe11db97d5 not found: ID does not exist" containerID="dbd820eca4751551dfb228fd7a960d1226759a1e733cd5108142b0fe11db97d5"
Jul 02 13:35:28.490913 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:28.490876    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbd820eca4751551dfb228fd7a960d1226759a1e733cd5108142b0fe11db97d5"} err="failed to get container status \"dbd820eca4751551dfb228fd7a960d1226759a1e733cd5108142b0fe11db97d5\": rpc error: code = NotFound desc = could not find container \"dbd820eca4751551dfb228fd7a960d1226759a1e733cd5108142b0fe11db97d5\": container with ID starting with dbd820eca4751551dfb228fd7a960d1226759a1e733cd5108142b0fe11db97d5 not found: ID does not exist"
Jul 02 13:35:28.504776 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:28.504744    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk"]
Jul 02 13:35:28.511612 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:28.511583    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk"]
Jul 02 13:35:28.717525 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:28.717468    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-nbwfk" podUID="9fd91d44-aa44-4a5d-9277-262ca1224372" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 02 13:35:29.233919 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:29.233883    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fd91d44-aa44-4a5d-9277-262ca1224372" path="/var/lib/kubelet/pods/9fd91d44-aa44-4a5d-9277-262ca1224372/volumes"
Jul 02 13:35:36.166660 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:36.166621    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:35:36.488576 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:36.488496    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2"]
Jul 02 13:35:36.488876 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:36.488862    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67907c8f-1d3f-4959-a69f-65f0eeee9693" containerName="main"
Jul 02 13:35:36.488929 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:36.488877    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="67907c8f-1d3f-4959-a69f-65f0eeee9693" containerName="main"
Jul 02 13:35:36.488929 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:36.488896    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fd91d44-aa44-4a5d-9277-262ca1224372" containerName="main"
Jul 02 13:35:36.488929 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:36.488902    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fd91d44-aa44-4a5d-9277-262ca1224372" containerName="main"
Jul 02 13:35:36.489038 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:36.488958    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="67907c8f-1d3f-4959-a69f-65f0eeee9693" containerName="main"
Jul 02 13:35:36.489038 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:36.488968    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fd91d44-aa44-4a5d-9277-262ca1224372" containerName="main"
Jul 02 13:35:36.493538 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:36.493513    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2"
Jul 02 13:35:36.497034 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:36.497016    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:35:36.497199 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:36.497180    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 02 13:35:36.497753 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:36.497738    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7s8sh\""
Jul 02 13:35:36.497819 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:36.497737    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-wpwzd\""
Jul 02 13:35:36.507805 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:36.507779    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2"]
Jul 02 13:35:36.517685 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:36.517642    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:35:36.599708 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:36.599667    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnl7q\" (UniqueName: \"kubernetes.io/projected/0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e-kube-api-access-lnl7q\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2\" (UID: \"0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2"
Jul 02 13:35:36.599917 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:36.599805    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2\" (UID: \"0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2"
Jul 02 13:35:36.599917 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:36.599879    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2\" (UID: \"0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2"
Jul 02 13:35:36.600025 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:36.599922    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2\" (UID: \"0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2"
Jul 02 13:35:36.701019 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:36.700983    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2\" (UID: \"0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2"
Jul 02 13:35:36.701228 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:36.701033    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2\" (UID: \"0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2"
Jul 02 13:35:36.701228 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:36.701055    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2\" (UID: \"0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2"
Jul 02 13:35:36.701228 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:36.701084    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnl7q\" (UniqueName: \"kubernetes.io/projected/0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e-kube-api-access-lnl7q\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2\" (UID: \"0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2"
Jul 02 13:35:36.701478 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:36.701451    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2\" (UID: \"0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2"
Jul 02 13:35:36.701563 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:36.701474    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2\" (UID: \"0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2"
Jul 02 13:35:36.703608 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:36.703589    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2\" (UID: \"0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2"
Jul 02 13:35:36.710114 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:36.710089    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnl7q\" (UniqueName: \"kubernetes.io/projected/0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e-kube-api-access-lnl7q\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2\" (UID: \"0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2"
Jul 02 13:35:36.804014 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:36.803919    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2"
Jul 02 13:35:36.944812 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:36.944777    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2"]
Jul 02 13:35:36.948213 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:35:36.948176    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb6c861_d0e2_4ce6_aa59_c39ea777ba5e.slice/crio-50b9610e6f5e478d4a4bc6844a98b1799148e5e9313e60b24dacc45c3c47bc7d WatchSource:0}: Error finding container 50b9610e6f5e478d4a4bc6844a98b1799148e5e9313e60b24dacc45c3c47bc7d: Status 404 returned error can't find the container with id 50b9610e6f5e478d4a4bc6844a98b1799148e5e9313e60b24dacc45c3c47bc7d
Jul 02 13:35:37.542057 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:37.542012    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2" event={"ID":"0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e","Type":"ContainerStarted","Data":"4c0f4b73f6a5fa4a7fc7fdf483d4bf8efe8d7d3a44ed7cd242a8ce3515290491"}
Jul 02 13:35:37.542578 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:37.542549    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2" event={"ID":"0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e","Type":"ContainerStarted","Data":"50b9610e6f5e478d4a4bc6844a98b1799148e5e9313e60b24dacc45c3c47bc7d"}
Jul 02 13:35:38.548774 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:38.548737    2583 generic.go:356] "Generic (PLEG): container finished" podID="0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e" containerID="4c0f4b73f6a5fa4a7fc7fdf483d4bf8efe8d7d3a44ed7cd242a8ce3515290491" exitCode=0
Jul 02 13:35:38.549149 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:38.548824    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2" event={"ID":"0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e","Type":"ContainerDied","Data":"4c0f4b73f6a5fa4a7fc7fdf483d4bf8efe8d7d3a44ed7cd242a8ce3515290491"}
Jul 02 13:35:39.555380 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:39.555343    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2" event={"ID":"0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e","Type":"ContainerStarted","Data":"5507e7431193f019d1457d8f611b4304f9f1e9d8e3e104a72e1da021bd288d8c"}
Jul 02 13:35:39.555779 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:39.555547    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2"
Jul 02 13:35:39.578054 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:39.577988    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2" podStartSLOduration=2.87407209 podStartE2EDuration="3.577967712s" totalImagesPullingTime="0s" totalInitContainerRuntime="703.895622ms" isStatefulPod=false podCreationTimestamp="2026-07-02 13:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:35:39.576856488 +0000 UTC m=+1250.804767663" watchObservedRunningTime="2026-07-02 13:35:39.577967712 +0000 UTC m=+1250.805878888"
Jul 02 13:35:40.810853 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:40.810755    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:35:57.244748 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:35:57.244695    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:36:04.010771 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:36:04.010743    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:36:04.072824 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:36:04.072786    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:36:10.562352 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:36:10.562316    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2"
Jul 02 13:36:31.027886 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:36:31.027821    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:36:43.472478 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:36:43.472440    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:37:05.924094 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:05.924064    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:37:06.115000 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:06.114962    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9"]
Jul 02 13:37:06.118651 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:06.118624    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9"
Jul 02 13:37:06.121450 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:06.121418    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Jul 02 13:37:06.121606 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:06.121475    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-gpg94\""
Jul 02 13:37:06.129262 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:06.129232    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9"]
Jul 02 13:37:06.147608 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:06.147569    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:37:06.225752 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:06.225660    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ec2dadf-bde2-4498-bdff-fdd7ee835cd6-kserve-provision-location\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9\" (UID: \"1ec2dadf-bde2-4498-bdff-fdd7ee835cd6\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9"
Jul 02 13:37:06.225752 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:06.225701    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcjmf\" (UniqueName: \"kubernetes.io/projected/1ec2dadf-bde2-4498-bdff-fdd7ee835cd6-kube-api-access-xcjmf\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9\" (UID: \"1ec2dadf-bde2-4498-bdff-fdd7ee835cd6\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9"
Jul 02 13:37:06.225947 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:06.225773    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ec2dadf-bde2-4498-bdff-fdd7ee835cd6-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9\" (UID: \"1ec2dadf-bde2-4498-bdff-fdd7ee835cd6\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9"
Jul 02 13:37:06.225947 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:06.225858    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1ec2dadf-bde2-4498-bdff-fdd7ee835cd6-tokenizer-uds\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9\" (UID: \"1ec2dadf-bde2-4498-bdff-fdd7ee835cd6\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9"
Jul 02 13:37:06.327327 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:06.327272    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1ec2dadf-bde2-4498-bdff-fdd7ee835cd6-tokenizer-uds\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9\" (UID: \"1ec2dadf-bde2-4498-bdff-fdd7ee835cd6\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9"
Jul 02 13:37:06.327573 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:06.327348    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ec2dadf-bde2-4498-bdff-fdd7ee835cd6-kserve-provision-location\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9\" (UID: \"1ec2dadf-bde2-4498-bdff-fdd7ee835cd6\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9"
Jul 02 13:37:06.327573 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:06.327367    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcjmf\" (UniqueName: \"kubernetes.io/projected/1ec2dadf-bde2-4498-bdff-fdd7ee835cd6-kube-api-access-xcjmf\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9\" (UID: \"1ec2dadf-bde2-4498-bdff-fdd7ee835cd6\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9"
Jul 02 13:37:06.327573 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:06.327435    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ec2dadf-bde2-4498-bdff-fdd7ee835cd6-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9\" (UID: \"1ec2dadf-bde2-4498-bdff-fdd7ee835cd6\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9"
Jul 02 13:37:06.327765 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:06.327740    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1ec2dadf-bde2-4498-bdff-fdd7ee835cd6-tokenizer-uds\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9\" (UID: \"1ec2dadf-bde2-4498-bdff-fdd7ee835cd6\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9"
Jul 02 13:37:06.327825 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:06.327770    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ec2dadf-bde2-4498-bdff-fdd7ee835cd6-kserve-provision-location\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9\" (UID: \"1ec2dadf-bde2-4498-bdff-fdd7ee835cd6\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9"
Jul 02 13:37:06.329881 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:06.329862    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ec2dadf-bde2-4498-bdff-fdd7ee835cd6-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9\" (UID: \"1ec2dadf-bde2-4498-bdff-fdd7ee835cd6\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9"
Jul 02 13:37:06.336993 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:06.336966    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcjmf\" (UniqueName: \"kubernetes.io/projected/1ec2dadf-bde2-4498-bdff-fdd7ee835cd6-kube-api-access-xcjmf\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9\" (UID: \"1ec2dadf-bde2-4498-bdff-fdd7ee835cd6\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9"
Jul 02 13:37:06.429547 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:06.429506    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9"
Jul 02 13:37:06.579401 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:06.579359    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9"]
Jul 02 13:37:06.581422 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:37:06.581369    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ec2dadf_bde2_4498_bdff_fdd7ee835cd6.slice/crio-92aa9df74f9631746fa17849375e48fd2a99cd70bb438fe861f4edc25693b683 WatchSource:0}: Error finding container 92aa9df74f9631746fa17849375e48fd2a99cd70bb438fe861f4edc25693b683: Status 404 returned error can't find the container with id 92aa9df74f9631746fa17849375e48fd2a99cd70bb438fe861f4edc25693b683
Jul 02 13:37:06.993633 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:06.993595    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9" event={"ID":"1ec2dadf-bde2-4498-bdff-fdd7ee835cd6","Type":"ContainerStarted","Data":"faf46973c9db7f779092c729362025c14bd8c949e7f5ff303aaac0d59e5ddbd9"}
Jul 02 13:37:06.994106 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:06.993649    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9" event={"ID":"1ec2dadf-bde2-4498-bdff-fdd7ee835cd6","Type":"ContainerStarted","Data":"92aa9df74f9631746fa17849375e48fd2a99cd70bb438fe861f4edc25693b683"}
Jul 02 13:37:08.000463 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:08.000430    2583 generic.go:356] "Generic (PLEG): container finished" podID="1ec2dadf-bde2-4498-bdff-fdd7ee835cd6" containerID="faf46973c9db7f779092c729362025c14bd8c949e7f5ff303aaac0d59e5ddbd9" exitCode=0
Jul 02 13:37:08.000882 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:08.000482    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9" event={"ID":"1ec2dadf-bde2-4498-bdff-fdd7ee835cd6","Type":"ContainerDied","Data":"faf46973c9db7f779092c729362025c14bd8c949e7f5ff303aaac0d59e5ddbd9"}
Jul 02 13:37:09.007580 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:09.007544    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9" event={"ID":"1ec2dadf-bde2-4498-bdff-fdd7ee835cd6","Type":"ContainerStarted","Data":"6adbfeafa5e4d661b068ba828741841f80281b805a146dd2d788f27034287f8c"}
Jul 02 13:37:09.007988 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:09.007640    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9"
Jul 02 13:37:09.029948 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:09.029898    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9" podStartSLOduration=2.333397936 podStartE2EDuration="3.02988276s" totalImagesPullingTime="0s" totalInitContainerRuntime="696.484824ms" isStatefulPod=false podCreationTimestamp="2026-07-02 13:37:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:37:09.028412978 +0000 UTC m=+1340.256324152" watchObservedRunningTime="2026-07-02 13:37:09.02988276 +0000 UTC m=+1340.257793934"
Jul 02 13:37:20.035855 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:20.035780    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:37:36.738117 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:36.738073    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:37:36.805684 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:36.805646    2583 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2" podUID="0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 02 13:37:36.830878 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:36.830845    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2"]
Jul 02 13:37:36.840029 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:37:36.839991    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 02 13:37:36.840029 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:37:36.840029 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:37:36.840029 ip-10-0-135-143 kubenswrapper[2583]:  > containerID="5507e7431193f019d1457d8f611b4304f9f1e9d8e3e104a72e1da021bd288d8c" cmd=["/bin/sleep","15"]
Jul 02 13:37:36.840316 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:36.840043    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2" message="" err=<
Jul 02 13:37:36.840316 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:37:36.840316 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:37:36.840316 ip-10-0-135-143 kubenswrapper[2583]:  >
Jul 02 13:37:36.840316 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:37:36.840057    2583 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 02 13:37:36.840316 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:37:36.840316 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:37:36.840316 ip-10-0-135-143 kubenswrapper[2583]:  > pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2" podUID="0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e" containerName="main" containerID="cri-o://5507e7431193f019d1457d8f611b4304f9f1e9d8e3e104a72e1da021bd288d8c"
Jul 02 13:37:36.840316 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:36.840111    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2" podUID="0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e" containerName="main" containerID="cri-o://5507e7431193f019d1457d8f611b4304f9f1e9d8e3e104a72e1da021bd288d8c" gracePeriod=60
Jul 02 13:37:36.875107 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:36.875075    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:37:37.092265 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:37.092239    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2"
Jul 02 13:37:37.153997 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:37.153964    2583 generic.go:356] "Generic (PLEG): container finished" podID="0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e" containerID="5507e7431193f019d1457d8f611b4304f9f1e9d8e3e104a72e1da021bd288d8c" exitCode=0
Jul 02 13:37:37.153997 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:37.154005    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2" event={"ID":"0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e","Type":"ContainerDied","Data":"5507e7431193f019d1457d8f611b4304f9f1e9d8e3e104a72e1da021bd288d8c"}
Jul 02 13:37:37.154240 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:37.154044    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2" event={"ID":"0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e","Type":"ContainerDied","Data":"50b9610e6f5e478d4a4bc6844a98b1799148e5e9313e60b24dacc45c3c47bc7d"}
Jul 02 13:37:37.154240 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:37.154063    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2"
Jul 02 13:37:37.154240 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:37.154066    2583 scope.go:117] "RemoveContainer" containerID="5507e7431193f019d1457d8f611b4304f9f1e9d8e3e104a72e1da021bd288d8c"
Jul 02 13:37:37.162845 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:37.162824    2583 scope.go:117] "RemoveContainer" containerID="4c0f4b73f6a5fa4a7fc7fdf483d4bf8efe8d7d3a44ed7cd242a8ce3515290491"
Jul 02 13:37:37.170840 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:37.170816    2583 scope.go:117] "RemoveContainer" containerID="5507e7431193f019d1457d8f611b4304f9f1e9d8e3e104a72e1da021bd288d8c"
Jul 02 13:37:37.171127 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:37:37.171104    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5507e7431193f019d1457d8f611b4304f9f1e9d8e3e104a72e1da021bd288d8c\": container with ID starting with 5507e7431193f019d1457d8f611b4304f9f1e9d8e3e104a72e1da021bd288d8c not found: ID does not exist" containerID="5507e7431193f019d1457d8f611b4304f9f1e9d8e3e104a72e1da021bd288d8c"
Jul 02 13:37:37.171175 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:37.171136    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5507e7431193f019d1457d8f611b4304f9f1e9d8e3e104a72e1da021bd288d8c"} err="failed to get container status \"5507e7431193f019d1457d8f611b4304f9f1e9d8e3e104a72e1da021bd288d8c\": rpc error: code = NotFound desc = could not find container \"5507e7431193f019d1457d8f611b4304f9f1e9d8e3e104a72e1da021bd288d8c\": container with ID starting with 5507e7431193f019d1457d8f611b4304f9f1e9d8e3e104a72e1da021bd288d8c not found: ID does not exist"
Jul 02 13:37:37.171175 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:37.171155    2583 scope.go:117] "RemoveContainer" containerID="4c0f4b73f6a5fa4a7fc7fdf483d4bf8efe8d7d3a44ed7cd242a8ce3515290491"
Jul 02 13:37:37.171364 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:37:37.171345    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c0f4b73f6a5fa4a7fc7fdf483d4bf8efe8d7d3a44ed7cd242a8ce3515290491\": container with ID starting with 4c0f4b73f6a5fa4a7fc7fdf483d4bf8efe8d7d3a44ed7cd242a8ce3515290491 not found: ID does not exist" containerID="4c0f4b73f6a5fa4a7fc7fdf483d4bf8efe8d7d3a44ed7cd242a8ce3515290491"
Jul 02 13:37:37.171430 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:37.171368    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c0f4b73f6a5fa4a7fc7fdf483d4bf8efe8d7d3a44ed7cd242a8ce3515290491"} err="failed to get container status \"4c0f4b73f6a5fa4a7fc7fdf483d4bf8efe8d7d3a44ed7cd242a8ce3515290491\": rpc error: code = NotFound desc = could not find container \"4c0f4b73f6a5fa4a7fc7fdf483d4bf8efe8d7d3a44ed7cd242a8ce3515290491\": container with ID starting with 4c0f4b73f6a5fa4a7fc7fdf483d4bf8efe8d7d3a44ed7cd242a8ce3515290491 not found: ID does not exist"
Jul 02 13:37:37.215366 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:37.215334    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e-tls-certs\") pod \"0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e\" (UID: \"0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e\") "
Jul 02 13:37:37.215366 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:37.215375    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnl7q\" (UniqueName: \"kubernetes.io/projected/0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e-kube-api-access-lnl7q\") pod \"0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e\" (UID: \"0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e\") "
Jul 02 13:37:37.215618 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:37.215455    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e-kserve-provision-location\") pod \"0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e\" (UID: \"0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e\") "
Jul 02 13:37:37.215618 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:37.215480    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e-tokenizer-uds\") pod \"0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e\" (UID: \"0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e\") "
Jul 02 13:37:37.215789 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:37.215761    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e" (UID: "0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:37:37.215887 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:37.215872    2583 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e-tokenizer-uds\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:37:37.216185 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:37.216160    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e" (UID: "0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:37:37.217545 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:37.217514    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e-kube-api-access-lnl7q" (OuterVolumeSpecName: "kube-api-access-lnl7q") pod "0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e" (UID: "0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e"). InnerVolumeSpecName "kube-api-access-lnl7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:37:37.217545 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:37.217524    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e" (UID: "0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:37:37.316767 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:37.316680    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:37:37.316767 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:37.316713    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnl7q\" (UniqueName: \"kubernetes.io/projected/0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e-kube-api-access-lnl7q\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:37:37.316767 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:37.316723    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e-kserve-provision-location\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:37:37.471373 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:37.471336    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2"]
Jul 02 13:37:37.474887 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:37.474857    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schegc4x2"]
Jul 02 13:37:39.233644 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:39.233608    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e" path="/var/lib/kubelet/pods/0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e/volumes"
Jul 02 13:37:40.015155 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:37:40.015125    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9"
Jul 02 13:38:02.070490 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:02.070451    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:38:02.246997 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:02.246961    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s"]
Jul 02 13:38:02.247352 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:02.247339    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e" containerName="storage-initializer"
Jul 02 13:38:02.247417 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:02.247354    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e" containerName="storage-initializer"
Jul 02 13:38:02.247417 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:02.247361    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e" containerName="main"
Jul 02 13:38:02.247417 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:02.247367    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e" containerName="main"
Jul 02 13:38:02.247522 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:02.247478    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fb6c861-d0e2-4ce6-aa59-c39ea777ba5e" containerName="main"
Jul 02 13:38:02.251603 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:02.251583    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s"
Jul 02 13:38:02.256009 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:02.255983    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-q9vrz\""
Jul 02 13:38:02.256177 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:02.256157    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 02 13:38:02.267881 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:02.267854    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s"]
Jul 02 13:38:02.287630 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:02.287592    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:38:02.348060 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:02.347957    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s\" (UID: \"8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s"
Jul 02 13:38:02.348060 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:02.348041    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s\" (UID: \"8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s"
Jul 02 13:38:02.348277 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:02.348081    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s\" (UID: \"8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s"
Jul 02 13:38:02.348277 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:02.348155    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njgkk\" (UniqueName: \"kubernetes.io/projected/8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8-kube-api-access-njgkk\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s\" (UID: \"8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s"
Jul 02 13:38:02.449644 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:02.449607    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s\" (UID: \"8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s"
Jul 02 13:38:02.449856 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:02.449659    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s\" (UID: \"8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s"
Jul 02 13:38:02.449856 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:02.449688    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s\" (UID: \"8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s"
Jul 02 13:38:02.449856 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:02.449717    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njgkk\" (UniqueName: \"kubernetes.io/projected/8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8-kube-api-access-njgkk\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s\" (UID: \"8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s"
Jul 02 13:38:02.450097 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:02.450076    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s\" (UID: \"8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s"
Jul 02 13:38:02.450166 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:02.450135    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s\" (UID: \"8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s"
Jul 02 13:38:02.452083 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:02.452062    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s\" (UID: \"8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s"
Jul 02 13:38:02.457969 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:02.457943    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njgkk\" (UniqueName: \"kubernetes.io/projected/8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8-kube-api-access-njgkk\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s\" (UID: \"8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s"
Jul 02 13:38:02.560858 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:02.560811    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s"
Jul 02 13:38:02.702019 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:02.701988    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s"]
Jul 02 13:38:02.704058 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:38:02.704028    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ae57bc0_868e_4fd0_a182_9f4abf9c7ae8.slice/crio-e64c1868fcfa3bd64a7345427267e0c7cdb13c25146242ff6f12fd7ecc97c80e WatchSource:0}: Error finding container e64c1868fcfa3bd64a7345427267e0c7cdb13c25146242ff6f12fd7ecc97c80e: Status 404 returned error can't find the container with id e64c1868fcfa3bd64a7345427267e0c7cdb13c25146242ff6f12fd7ecc97c80e
Jul 02 13:38:03.279068 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:03.279032    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s" event={"ID":"8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8","Type":"ContainerStarted","Data":"7e744a18543e3e671e0500d1aa093ba1194b3d5e7671514b77b676d442c2d7e1"}
Jul 02 13:38:03.279466 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:03.279081    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s" event={"ID":"8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8","Type":"ContainerStarted","Data":"e64c1868fcfa3bd64a7345427267e0c7cdb13c25146242ff6f12fd7ecc97c80e"}
Jul 02 13:38:04.284764 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:04.284719    2583 generic.go:356] "Generic (PLEG): container finished" podID="8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8" containerID="7e744a18543e3e671e0500d1aa093ba1194b3d5e7671514b77b676d442c2d7e1" exitCode=0
Jul 02 13:38:04.285143 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:04.284807    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s" event={"ID":"8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8","Type":"ContainerDied","Data":"7e744a18543e3e671e0500d1aa093ba1194b3d5e7671514b77b676d442c2d7e1"}
Jul 02 13:38:05.292607 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:05.292572    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s" event={"ID":"8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8","Type":"ContainerStarted","Data":"7370966b80540631ade14830e44449010a4b3c2ca1cc7312001f266ae04dc329"}
Jul 02 13:38:05.292989 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:05.292709    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s"
Jul 02 13:38:05.316487 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:05.316432    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s" podStartSLOduration=2.633658625 podStartE2EDuration="3.316415627s" totalImagesPullingTime="0s" totalInitContainerRuntime="682.757002ms" isStatefulPod=false podCreationTimestamp="2026-07-02 13:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:38:05.314720494 +0000 UTC m=+1396.542631662" watchObservedRunningTime="2026-07-02 13:38:05.316415627 +0000 UTC m=+1396.544326801"
Jul 02 13:38:36.299284 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:36.299252    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s"
Jul 02 13:38:46.138009 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:38:46.137914    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:39:21.104545 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:21.104508    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9"]
Jul 02 13:39:21.114317 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:39:21.114270    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 02 13:39:21.114317 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:39:21.114317 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:39:21.114317 ip-10-0-135-143 kubenswrapper[2583]:  > containerID="6adbfeafa5e4d661b068ba828741841f80281b805a146dd2d788f27034287f8c" cmd=["/bin/sleep","15"]
Jul 02 13:39:21.114673 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:21.114336    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9" message="" err=<
Jul 02 13:39:21.114673 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:39:21.114673 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:39:21.114673 ip-10-0-135-143 kubenswrapper[2583]:  >
Jul 02 13:39:21.114673 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:39:21.114351    2583 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 02 13:39:21.114673 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:39:21.114673 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:39:21.114673 ip-10-0-135-143 kubenswrapper[2583]:  > pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9" podUID="1ec2dadf-bde2-4498-bdff-fdd7ee835cd6" containerName="main" containerID="cri-o://6adbfeafa5e4d661b068ba828741841f80281b805a146dd2d788f27034287f8c"
Jul 02 13:39:21.114673 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:21.114406    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9" podUID="1ec2dadf-bde2-4498-bdff-fdd7ee835cd6" containerName="main" containerID="cri-o://6adbfeafa5e4d661b068ba828741841f80281b805a146dd2d788f27034287f8c" gracePeriod=60
Jul 02 13:39:21.369663 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:21.369631    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9"
Jul 02 13:39:21.479650 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:21.479618    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ec2dadf-bde2-4498-bdff-fdd7ee835cd6-tls-certs\") pod \"1ec2dadf-bde2-4498-bdff-fdd7ee835cd6\" (UID: \"1ec2dadf-bde2-4498-bdff-fdd7ee835cd6\") "
Jul 02 13:39:21.479845 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:21.479727    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcjmf\" (UniqueName: \"kubernetes.io/projected/1ec2dadf-bde2-4498-bdff-fdd7ee835cd6-kube-api-access-xcjmf\") pod \"1ec2dadf-bde2-4498-bdff-fdd7ee835cd6\" (UID: \"1ec2dadf-bde2-4498-bdff-fdd7ee835cd6\") "
Jul 02 13:39:21.479845 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:21.479751    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ec2dadf-bde2-4498-bdff-fdd7ee835cd6-kserve-provision-location\") pod \"1ec2dadf-bde2-4498-bdff-fdd7ee835cd6\" (UID: \"1ec2dadf-bde2-4498-bdff-fdd7ee835cd6\") "
Jul 02 13:39:21.479845 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:21.479780    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1ec2dadf-bde2-4498-bdff-fdd7ee835cd6-tokenizer-uds\") pod \"1ec2dadf-bde2-4498-bdff-fdd7ee835cd6\" (UID: \"1ec2dadf-bde2-4498-bdff-fdd7ee835cd6\") "
Jul 02 13:39:21.480106 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:21.480076    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ec2dadf-bde2-4498-bdff-fdd7ee835cd6-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "1ec2dadf-bde2-4498-bdff-fdd7ee835cd6" (UID: "1ec2dadf-bde2-4498-bdff-fdd7ee835cd6"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:39:21.480477 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:21.480454    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ec2dadf-bde2-4498-bdff-fdd7ee835cd6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1ec2dadf-bde2-4498-bdff-fdd7ee835cd6" (UID: "1ec2dadf-bde2-4498-bdff-fdd7ee835cd6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:39:21.481736 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:21.481716    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ec2dadf-bde2-4498-bdff-fdd7ee835cd6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1ec2dadf-bde2-4498-bdff-fdd7ee835cd6" (UID: "1ec2dadf-bde2-4498-bdff-fdd7ee835cd6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:39:21.481881 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:21.481862    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec2dadf-bde2-4498-bdff-fdd7ee835cd6-kube-api-access-xcjmf" (OuterVolumeSpecName: "kube-api-access-xcjmf") pod "1ec2dadf-bde2-4498-bdff-fdd7ee835cd6" (UID: "1ec2dadf-bde2-4498-bdff-fdd7ee835cd6"). InnerVolumeSpecName "kube-api-access-xcjmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:39:21.580451 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:21.580376    2583 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1ec2dadf-bde2-4498-bdff-fdd7ee835cd6-tokenizer-uds\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:39:21.580451 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:21.580447    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ec2dadf-bde2-4498-bdff-fdd7ee835cd6-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:39:21.580451 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:21.580457    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcjmf\" (UniqueName: \"kubernetes.io/projected/1ec2dadf-bde2-4498-bdff-fdd7ee835cd6-kube-api-access-xcjmf\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:39:21.580703 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:21.580467    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ec2dadf-bde2-4498-bdff-fdd7ee835cd6-kserve-provision-location\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:39:21.669380 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:21.669286    2583 generic.go:356] "Generic (PLEG): container finished" podID="1ec2dadf-bde2-4498-bdff-fdd7ee835cd6" containerID="6adbfeafa5e4d661b068ba828741841f80281b805a146dd2d788f27034287f8c" exitCode=0
Jul 02 13:39:21.669380 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:21.669333    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9" event={"ID":"1ec2dadf-bde2-4498-bdff-fdd7ee835cd6","Type":"ContainerDied","Data":"6adbfeafa5e4d661b068ba828741841f80281b805a146dd2d788f27034287f8c"}
Jul 02 13:39:21.669380 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:21.669379    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9" event={"ID":"1ec2dadf-bde2-4498-bdff-fdd7ee835cd6","Type":"ContainerDied","Data":"92aa9df74f9631746fa17849375e48fd2a99cd70bb438fe861f4edc25693b683"}
Jul 02 13:39:21.669692 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:21.669339    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9"
Jul 02 13:39:21.669692 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:21.669426    2583 scope.go:117] "RemoveContainer" containerID="6adbfeafa5e4d661b068ba828741841f80281b805a146dd2d788f27034287f8c"
Jul 02 13:39:21.679189 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:21.679168    2583 scope.go:117] "RemoveContainer" containerID="faf46973c9db7f779092c729362025c14bd8c949e7f5ff303aaac0d59e5ddbd9"
Jul 02 13:39:21.688357 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:21.688335    2583 scope.go:117] "RemoveContainer" containerID="6adbfeafa5e4d661b068ba828741841f80281b805a146dd2d788f27034287f8c"
Jul 02 13:39:21.688769 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:39:21.688735    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6adbfeafa5e4d661b068ba828741841f80281b805a146dd2d788f27034287f8c\": container with ID starting with 6adbfeafa5e4d661b068ba828741841f80281b805a146dd2d788f27034287f8c not found: ID does not exist" containerID="6adbfeafa5e4d661b068ba828741841f80281b805a146dd2d788f27034287f8c"
Jul 02 13:39:21.689181 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:21.688779    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6adbfeafa5e4d661b068ba828741841f80281b805a146dd2d788f27034287f8c"} err="failed to get container status \"6adbfeafa5e4d661b068ba828741841f80281b805a146dd2d788f27034287f8c\": rpc error: code = NotFound desc = could not find container \"6adbfeafa5e4d661b068ba828741841f80281b805a146dd2d788f27034287f8c\": container with ID starting with 6adbfeafa5e4d661b068ba828741841f80281b805a146dd2d788f27034287f8c not found: ID does not exist"
Jul 02 13:39:21.689181 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:21.688805    2583 scope.go:117] "RemoveContainer" containerID="faf46973c9db7f779092c729362025c14bd8c949e7f5ff303aaac0d59e5ddbd9"
Jul 02 13:39:21.689181 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:39:21.689167    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faf46973c9db7f779092c729362025c14bd8c949e7f5ff303aaac0d59e5ddbd9\": container with ID starting with faf46973c9db7f779092c729362025c14bd8c949e7f5ff303aaac0d59e5ddbd9 not found: ID does not exist" containerID="faf46973c9db7f779092c729362025c14bd8c949e7f5ff303aaac0d59e5ddbd9"
Jul 02 13:39:21.689406 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:21.689198    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faf46973c9db7f779092c729362025c14bd8c949e7f5ff303aaac0d59e5ddbd9"} err="failed to get container status \"faf46973c9db7f779092c729362025c14bd8c949e7f5ff303aaac0d59e5ddbd9\": rpc error: code = NotFound desc = could not find container \"faf46973c9db7f779092c729362025c14bd8c949e7f5ff303aaac0d59e5ddbd9\": container with ID starting with faf46973c9db7f779092c729362025c14bd8c949e7f5ff303aaac0d59e5ddbd9 not found: ID does not exist"
Jul 02 13:39:21.691252 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:21.691229    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9"]
Jul 02 13:39:21.696181 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:21.696153    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheq48p9"]
Jul 02 13:39:22.529848 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:22.529811    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:39:22.584596 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:22.584563    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:39:23.233875 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:23.233844    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ec2dadf-bde2-4498-bdff-fdd7ee835cd6" path="/var/lib/kubelet/pods/1ec2dadf-bde2-4498-bdff-fdd7ee835cd6/volumes"
Jul 02 13:39:33.732253 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:33.732215    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s"]
Jul 02 13:39:33.740793 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:33.740754    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:39:33.742470 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:39:33.742438    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 02 13:39:33.742470 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:39:33.742470 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:39:33.742470 ip-10-0-135-143 kubenswrapper[2583]:  > containerID="7370966b80540631ade14830e44449010a4b3c2ca1cc7312001f266ae04dc329" cmd=["/bin/sleep","15"]
Jul 02 13:39:33.742663 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:33.742482    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s" message="" err=<
Jul 02 13:39:33.742663 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:39:33.742663 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:39:33.742663 ip-10-0-135-143 kubenswrapper[2583]:  >
Jul 02 13:39:33.742663 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:39:33.742494    2583 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 02 13:39:33.742663 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:39:33.742663 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:39:33.742663 ip-10-0-135-143 kubenswrapper[2583]:  > pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s" podUID="8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8" containerName="main" containerID="cri-o://7370966b80540631ade14830e44449010a4b3c2ca1cc7312001f266ae04dc329"
Jul 02 13:39:33.742663 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:33.742538    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s" podUID="8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8" containerName="main" containerID="cri-o://7370966b80540631ade14830e44449010a4b3c2ca1cc7312001f266ae04dc329" gracePeriod=60
Jul 02 13:39:33.812736 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:33.812690    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:39:33.996858 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:33.996795    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s"
Jul 02 13:39:34.094582 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:34.094545    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njgkk\" (UniqueName: \"kubernetes.io/projected/8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8-kube-api-access-njgkk\") pod \"8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8\" (UID: \"8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8\") "
Jul 02 13:39:34.094741 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:34.094609    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8-tokenizer-uds\") pod \"8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8\" (UID: \"8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8\") "
Jul 02 13:39:34.094741 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:34.094680    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8-kserve-provision-location\") pod \"8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8\" (UID: \"8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8\") "
Jul 02 13:39:34.094741 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:34.094720    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8-tls-certs\") pod \"8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8\" (UID: \"8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8\") "
Jul 02 13:39:34.094907 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:34.094885    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8" (UID: "8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:39:34.095043 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:34.095028    2583 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8-tokenizer-uds\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:39:34.095320 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:34.095291    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8" (UID: "8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:39:34.096703 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:34.096676    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8-kube-api-access-njgkk" (OuterVolumeSpecName: "kube-api-access-njgkk") pod "8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8" (UID: "8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8"). InnerVolumeSpecName "kube-api-access-njgkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:39:34.096873 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:34.096857    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8" (UID: "8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:39:34.195801 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:34.195766    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:39:34.195801 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:34.195802    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njgkk\" (UniqueName: \"kubernetes.io/projected/8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8-kube-api-access-njgkk\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:39:34.196024 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:34.195812    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8-kserve-provision-location\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:39:34.737975 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:34.737942    2583 generic.go:356] "Generic (PLEG): container finished" podID="8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8" containerID="7370966b80540631ade14830e44449010a4b3c2ca1cc7312001f266ae04dc329" exitCode=0
Jul 02 13:39:34.738426 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:34.737993    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s"
Jul 02 13:39:34.738426 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:34.738009    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s" event={"ID":"8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8","Type":"ContainerDied","Data":"7370966b80540631ade14830e44449010a4b3c2ca1cc7312001f266ae04dc329"}
Jul 02 13:39:34.738426 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:34.738051    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s" event={"ID":"8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8","Type":"ContainerDied","Data":"e64c1868fcfa3bd64a7345427267e0c7cdb13c25146242ff6f12fd7ecc97c80e"}
Jul 02 13:39:34.738426 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:34.738072    2583 scope.go:117] "RemoveContainer" containerID="7370966b80540631ade14830e44449010a4b3c2ca1cc7312001f266ae04dc329"
Jul 02 13:39:34.747593 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:34.747464    2583 scope.go:117] "RemoveContainer" containerID="7e744a18543e3e671e0500d1aa093ba1194b3d5e7671514b77b676d442c2d7e1"
Jul 02 13:39:34.755428 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:34.755401    2583 scope.go:117] "RemoveContainer" containerID="7370966b80540631ade14830e44449010a4b3c2ca1cc7312001f266ae04dc329"
Jul 02 13:39:34.755701 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:39:34.755681    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7370966b80540631ade14830e44449010a4b3c2ca1cc7312001f266ae04dc329\": container with ID starting with 7370966b80540631ade14830e44449010a4b3c2ca1cc7312001f266ae04dc329 not found: ID does not exist" containerID="7370966b80540631ade14830e44449010a4b3c2ca1cc7312001f266ae04dc329"
Jul 02 13:39:34.755754 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:34.755710    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7370966b80540631ade14830e44449010a4b3c2ca1cc7312001f266ae04dc329"} err="failed to get container status \"7370966b80540631ade14830e44449010a4b3c2ca1cc7312001f266ae04dc329\": rpc error: code = NotFound desc = could not find container \"7370966b80540631ade14830e44449010a4b3c2ca1cc7312001f266ae04dc329\": container with ID starting with 7370966b80540631ade14830e44449010a4b3c2ca1cc7312001f266ae04dc329 not found: ID does not exist"
Jul 02 13:39:34.755754 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:34.755727    2583 scope.go:117] "RemoveContainer" containerID="7e744a18543e3e671e0500d1aa093ba1194b3d5e7671514b77b676d442c2d7e1"
Jul 02 13:39:34.755964 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:39:34.755946    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e744a18543e3e671e0500d1aa093ba1194b3d5e7671514b77b676d442c2d7e1\": container with ID starting with 7e744a18543e3e671e0500d1aa093ba1194b3d5e7671514b77b676d442c2d7e1 not found: ID does not exist" containerID="7e744a18543e3e671e0500d1aa093ba1194b3d5e7671514b77b676d442c2d7e1"
Jul 02 13:39:34.756009 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:34.755971    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e744a18543e3e671e0500d1aa093ba1194b3d5e7671514b77b676d442c2d7e1"} err="failed to get container status \"7e744a18543e3e671e0500d1aa093ba1194b3d5e7671514b77b676d442c2d7e1\": rpc error: code = NotFound desc = could not find container \"7e744a18543e3e671e0500d1aa093ba1194b3d5e7671514b77b676d442c2d7e1\": container with ID starting with 7e744a18543e3e671e0500d1aa093ba1194b3d5e7671514b77b676d442c2d7e1 not found: ID does not exist"
Jul 02 13:39:34.760469 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:34.760444    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s"]
Jul 02 13:39:34.764007 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:34.763980    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh6f2s"]
Jul 02 13:39:35.234305 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:35.234272    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8" path="/var/lib/kubelet/pods/8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8/volumes"
Jul 02 13:39:49.318154 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:49.318124    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 13:39:49.321986 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:49.321959    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 13:39:50.051080 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:50.051045    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:39:50.521239 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:50.521201    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj"]
Jul 02 13:39:50.521802 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:50.521784    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ec2dadf-bde2-4498-bdff-fdd7ee835cd6" containerName="storage-initializer"
Jul 02 13:39:50.521861 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:50.521805    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ec2dadf-bde2-4498-bdff-fdd7ee835cd6" containerName="storage-initializer"
Jul 02 13:39:50.521861 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:50.521817    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8" containerName="main"
Jul 02 13:39:50.521861 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:50.521826    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8" containerName="main"
Jul 02 13:39:50.521861 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:50.521854    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8" containerName="storage-initializer"
Jul 02 13:39:50.521990 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:50.521863    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8" containerName="storage-initializer"
Jul 02 13:39:50.521990 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:50.521877    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ec2dadf-bde2-4498-bdff-fdd7ee835cd6" containerName="main"
Jul 02 13:39:50.521990 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:50.521885    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ec2dadf-bde2-4498-bdff-fdd7ee835cd6" containerName="main"
Jul 02 13:39:50.521990 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:50.521970    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ae57bc0-868e-4fd0-a182-9f4abf9c7ae8" containerName="main"
Jul 02 13:39:50.521990 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:50.521982    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ec2dadf-bde2-4498-bdff-fdd7ee835cd6" containerName="main"
Jul 02 13:39:50.526789 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:50.526765    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj"
Jul 02 13:39:50.530757 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:50.530734    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7s8sh\""
Jul 02 13:39:50.530899 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:50.530737    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:39:50.530899 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:50.530739    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-epp-sa-dockercfg-vlqt9\""
Jul 02 13:39:50.530899 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:50.530824    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jul 02 13:39:50.535756 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:50.535731    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj"]
Jul 02 13:39:50.562169 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:50.562136    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:39:50.649402 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:50.649361    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds8q2\" (UniqueName: \"kubernetes.io/projected/9a92ddaf-d371-4a7c-9772-780ca0efe46c-kube-api-access-ds8q2\") pod \"custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj\" (UID: \"9a92ddaf-d371-4a7c-9772-780ca0efe46c\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj"
Jul 02 13:39:50.649584 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:50.649433    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9a92ddaf-d371-4a7c-9772-780ca0efe46c-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj\" (UID: \"9a92ddaf-d371-4a7c-9772-780ca0efe46c\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj"
Jul 02 13:39:50.649584 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:50.649454    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a92ddaf-d371-4a7c-9772-780ca0efe46c-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj\" (UID: \"9a92ddaf-d371-4a7c-9772-780ca0efe46c\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj"
Jul 02 13:39:50.649584 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:50.649553    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a92ddaf-d371-4a7c-9772-780ca0efe46c-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj\" (UID: \"9a92ddaf-d371-4a7c-9772-780ca0efe46c\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj"
Jul 02 13:39:50.750822 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:50.750785    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds8q2\" (UniqueName: \"kubernetes.io/projected/9a92ddaf-d371-4a7c-9772-780ca0efe46c-kube-api-access-ds8q2\") pod \"custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj\" (UID: \"9a92ddaf-d371-4a7c-9772-780ca0efe46c\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj"
Jul 02 13:39:50.750822 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:50.750822    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9a92ddaf-d371-4a7c-9772-780ca0efe46c-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj\" (UID: \"9a92ddaf-d371-4a7c-9772-780ca0efe46c\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj"
Jul 02 13:39:50.751077 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:50.750842    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a92ddaf-d371-4a7c-9772-780ca0efe46c-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj\" (UID: \"9a92ddaf-d371-4a7c-9772-780ca0efe46c\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj"
Jul 02 13:39:50.751077 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:50.750910    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a92ddaf-d371-4a7c-9772-780ca0efe46c-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj\" (UID: \"9a92ddaf-d371-4a7c-9772-780ca0efe46c\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj"
Jul 02 13:39:50.751312 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:50.751278    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a92ddaf-d371-4a7c-9772-780ca0efe46c-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj\" (UID: \"9a92ddaf-d371-4a7c-9772-780ca0efe46c\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj"
Jul 02 13:39:50.751450 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:50.751293    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9a92ddaf-d371-4a7c-9772-780ca0efe46c-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj\" (UID: \"9a92ddaf-d371-4a7c-9772-780ca0efe46c\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj"
Jul 02 13:39:50.753269 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:50.753251    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a92ddaf-d371-4a7c-9772-780ca0efe46c-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj\" (UID: \"9a92ddaf-d371-4a7c-9772-780ca0efe46c\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj"
Jul 02 13:39:50.759195 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:50.759169    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds8q2\" (UniqueName: \"kubernetes.io/projected/9a92ddaf-d371-4a7c-9772-780ca0efe46c-kube-api-access-ds8q2\") pod \"custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj\" (UID: \"9a92ddaf-d371-4a7c-9772-780ca0efe46c\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj"
Jul 02 13:39:50.837091 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:50.836991    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj"
Jul 02 13:39:50.974646 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:50.974619    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj"]
Jul 02 13:39:50.976956 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:39:50.976927    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a92ddaf_d371_4a7c_9772_780ca0efe46c.slice/crio-628a57f16a793cdec7d5ee586684ddf39524e82b4e68fe62c0a82d96bdb6a934 WatchSource:0}: Error finding container 628a57f16a793cdec7d5ee586684ddf39524e82b4e68fe62c0a82d96bdb6a934: Status 404 returned error can't find the container with id 628a57f16a793cdec7d5ee586684ddf39524e82b4e68fe62c0a82d96bdb6a934
Jul 02 13:39:51.727850 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:39:51.727805    2583 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a92ddaf_d371_4a7c_9772_780ca0efe46c.slice/crio-conmon-4723cbbed0b9cc0e4de93b41ed40498816a13f2c437e5e5a7fc2b0a0b53753ee.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a92ddaf_d371_4a7c_9772_780ca0efe46c.slice/crio-4723cbbed0b9cc0e4de93b41ed40498816a13f2c437e5e5a7fc2b0a0b53753ee.scope\": RecentStats: unable to find data in memory cache]"
Jul 02 13:39:51.821088 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:51.821054    2583 generic.go:356] "Generic (PLEG): container finished" podID="9a92ddaf-d371-4a7c-9772-780ca0efe46c" containerID="4723cbbed0b9cc0e4de93b41ed40498816a13f2c437e5e5a7fc2b0a0b53753ee" exitCode=0
Jul 02 13:39:51.821344 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:51.821147    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj" event={"ID":"9a92ddaf-d371-4a7c-9772-780ca0efe46c","Type":"ContainerDied","Data":"4723cbbed0b9cc0e4de93b41ed40498816a13f2c437e5e5a7fc2b0a0b53753ee"}
Jul 02 13:39:51.821344 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:51.821203    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj" event={"ID":"9a92ddaf-d371-4a7c-9772-780ca0efe46c","Type":"ContainerStarted","Data":"628a57f16a793cdec7d5ee586684ddf39524e82b4e68fe62c0a82d96bdb6a934"}
Jul 02 13:39:52.827554 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:52.827519    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj" event={"ID":"9a92ddaf-d371-4a7c-9772-780ca0efe46c","Type":"ContainerStarted","Data":"20a187fe6a02963b156155bed4ad7336a32eccb851dd30c6e761f115aa7459c1"}
Jul 02 13:39:52.827951 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:52.827643    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj"
Jul 02 13:39:52.848274 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:52.848211    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj" podStartSLOduration=2.198913944 podStartE2EDuration="2.84819364s" totalImagesPullingTime="0s" totalInitContainerRuntime="649.279696ms" isStatefulPod=false podCreationTimestamp="2026-07-02 13:39:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:39:52.848093686 +0000 UTC m=+1504.076004861" watchObservedRunningTime="2026-07-02 13:39:52.84819364 +0000 UTC m=+1504.076104814"
Jul 02 13:39:53.972354 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:53.972315    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:39:54.072781 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:54.072740    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:39:54.129338 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:54.129305    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:39:56.520742 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:56.520702    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:39:56.801433 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:56.801326    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:39:57.038248 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:57.038206    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-9pg7p"]
Jul 02 13:39:57.041926 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:57.041901    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-9pg7p"
Jul 02 13:39:57.049275 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:57.049253    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-9pg7p"]
Jul 02 13:39:57.096647 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:57.096561    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:39:57.103032 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:57.103005    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nt27\" (UniqueName: \"kubernetes.io/projected/970f3494-f719-471d-818d-04f90719f2a3-kube-api-access-2nt27\") pod \"e2e-pvc-model-download-9pg7p\" (UID: \"970f3494-f719-471d-818d-04f90719f2a3\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-9pg7p"
Jul 02 13:39:57.204433 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:57.204369    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nt27\" (UniqueName: \"kubernetes.io/projected/970f3494-f719-471d-818d-04f90719f2a3-kube-api-access-2nt27\") pod \"e2e-pvc-model-download-9pg7p\" (UID: \"970f3494-f719-471d-818d-04f90719f2a3\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-9pg7p"
Jul 02 13:39:57.212963 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:57.212932    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nt27\" (UniqueName: \"kubernetes.io/projected/970f3494-f719-471d-818d-04f90719f2a3-kube-api-access-2nt27\") pod \"e2e-pvc-model-download-9pg7p\" (UID: \"970f3494-f719-471d-818d-04f90719f2a3\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-9pg7p"
Jul 02 13:39:58.403569 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:39:58.403541    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:40:00.939989 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:40:00.939952    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-44641bc5-8148-4920-9d29-b75c22fd719a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b6542df029b7bcd7\") pod \"e2e-pvc-model-download-9pg7p\" (UID: \"970f3494-f719-471d-818d-04f90719f2a3\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-9pg7p"
Jul 02 13:40:00.945109 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:40:00.945084    2583 operation_generator.go:1469] "Controller attach succeeded for volume \"pvc-44641bc5-8148-4920-9d29-b75c22fd719a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b6542df029b7bcd7\") pod \"e2e-pvc-model-download-9pg7p\" (UID: \"970f3494-f719-471d-818d-04f90719f2a3\") device path: \"\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-9pg7p"
Jul 02 13:40:01.041458 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:40:01.041415    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-44641bc5-8148-4920-9d29-b75c22fd719a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b6542df029b7bcd7\") pod \"e2e-pvc-model-download-9pg7p\" (UID: \"970f3494-f719-471d-818d-04f90719f2a3\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-9pg7p"
Jul 02 13:40:01.041629 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:40:01.041571    2583 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-44641bc5-8148-4920-9d29-b75c22fd719a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b6542df029b7bcd7\") pod \"e2e-pvc-model-download-9pg7p\" (UID: \"970f3494-f719-471d-818d-04f90719f2a3\") DevicePath \"\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-9pg7p"
Jul 02 13:40:01.044409 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:40:01.044364    2583 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-44641bc5-8148-4920-9d29-b75c22fd719a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b6542df029b7bcd7\") pod \"e2e-pvc-model-download-9pg7p\" (UID: \"970f3494-f719-471d-818d-04f90719f2a3\") DevicePath \"csi-1a61f6244756a76c303f7b15014f85abfa34b9b235be89806913394e2ab12862\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-9pg7p"
Jul 02 13:40:01.260606 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:40:01.260576    2583 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-44641bc5-8148-4920-9d29-b75c22fd719a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b6542df029b7bcd7\") pod \"e2e-pvc-model-download-9pg7p\" (UID: \"970f3494-f719-471d-818d-04f90719f2a3\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/86c6b61c91c4d06e5938fc2dd8dea7e3e98bb85489e27e6fc0857c958e8ae788/globalmount\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-9pg7p"
Jul 02 13:40:01.274457 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:40:01.274425    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-44641bc5-8148-4920-9d29-b75c22fd719a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b6542df029b7bcd7\") pod \"e2e-pvc-model-download-9pg7p\" (UID: \"970f3494-f719-471d-818d-04f90719f2a3\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-9pg7p"
Jul 02 13:40:01.551564 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:40:01.551482    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-9pg7p"
Jul 02 13:40:01.682626 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:40:01.682598    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-9pg7p"]
Jul 02 13:40:01.684819 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:40:01.684788    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod970f3494_f719_471d_818d_04f90719f2a3.slice/crio-41852c03444f993ee800af829b0b67a8e2641ebc8c5acd715132ac15f16c3d7f WatchSource:0}: Error finding container 41852c03444f993ee800af829b0b67a8e2641ebc8c5acd715132ac15f16c3d7f: Status 404 returned error can't find the container with id 41852c03444f993ee800af829b0b67a8e2641ebc8c5acd715132ac15f16c3d7f
Jul 02 13:40:01.873293 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:40:01.873251    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-9pg7p" event={"ID":"970f3494-f719-471d-818d-04f90719f2a3","Type":"ContainerStarted","Data":"2ec32ebca0c27382f1a31b83e52ee91988ce8600b162e4fa3918a296bcae16c3"}
Jul 02 13:40:01.873534 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:40:01.873300    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-9pg7p" event={"ID":"970f3494-f719-471d-818d-04f90719f2a3","Type":"ContainerStarted","Data":"41852c03444f993ee800af829b0b67a8e2641ebc8c5acd715132ac15f16c3d7f"}
Jul 02 13:40:01.889850 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:40:01.889791    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/e2e-pvc-model-download-9pg7p" podStartSLOduration=7.889769528 podStartE2EDuration="7.889769528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:39:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:40:01.887945286 +0000 UTC m=+1513.115856460" watchObservedRunningTime="2026-07-02 13:40:01.889769528 +0000 UTC m=+1513.117680703"
Jul 02 13:40:08.917130 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:40:08.917089    2583 generic.go:356] "Generic (PLEG): container finished" podID="970f3494-f719-471d-818d-04f90719f2a3" containerID="2ec32ebca0c27382f1a31b83e52ee91988ce8600b162e4fa3918a296bcae16c3" exitCode=0
Jul 02 13:40:08.917595 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:40:08.917151    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-9pg7p" event={"ID":"970f3494-f719-471d-818d-04f90719f2a3","Type":"ContainerDied","Data":"2ec32ebca0c27382f1a31b83e52ee91988ce8600b162e4fa3918a296bcae16c3"}
Jul 02 13:40:10.079283 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:40:10.079251    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-9pg7p"
Jul 02 13:40:10.124320 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:40:10.124285    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nt27\" (UniqueName: \"kubernetes.io/projected/970f3494-f719-471d-818d-04f90719f2a3-kube-api-access-2nt27\") pod \"970f3494-f719-471d-818d-04f90719f2a3\" (UID: \"970f3494-f719-471d-818d-04f90719f2a3\") "
Jul 02 13:40:10.124501 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:40:10.124441    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b6542df029b7bcd7\") pod \"970f3494-f719-471d-818d-04f90719f2a3\" (UID: \"970f3494-f719-471d-818d-04f90719f2a3\") "
Jul 02 13:40:10.126459 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:40:10.126425    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/970f3494-f719-471d-818d-04f90719f2a3-kube-api-access-2nt27" (OuterVolumeSpecName: "kube-api-access-2nt27") pod "970f3494-f719-471d-818d-04f90719f2a3" (UID: "970f3494-f719-471d-818d-04f90719f2a3"). InnerVolumeSpecName "kube-api-access-2nt27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:40:10.128296 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:40:10.128275    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0b6542df029b7bcd7" (OuterVolumeSpecName: "model-storage") pod "970f3494-f719-471d-818d-04f90719f2a3" (UID: "970f3494-f719-471d-818d-04f90719f2a3"). InnerVolumeSpecName "pvc-44641bc5-8148-4920-9d29-b75c22fd719a". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 02 13:40:10.226174 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:40:10.226081    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nt27\" (UniqueName: \"kubernetes.io/projected/970f3494-f719-471d-818d-04f90719f2a3-kube-api-access-2nt27\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:40:10.226174 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:40:10.226136    2583 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-44641bc5-8148-4920-9d29-b75c22fd719a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b6542df029b7bcd7\") on node \"ip-10-0-135-143.ec2.internal\" "
Jul 02 13:40:10.334428 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:40:10.334402    2583 operation_generator.go:895] UnmountDevice succeeded for volume "pvc-44641bc5-8148-4920-9d29-b75c22fd719a" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0b6542df029b7bcd7") on node "ip-10-0-135-143.ec2.internal"
Jul 02 13:40:10.427836 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:40:10.427777    2583 reconciler_common.go:299] "Volume detached for volume \"pvc-44641bc5-8148-4920-9d29-b75c22fd719a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b6542df029b7bcd7\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"csi-1a61f6244756a76c303f7b15014f85abfa34b9b235be89806913394e2ab12862\""
Jul 02 13:40:10.928523 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:40:10.928446    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-9pg7p"
Jul 02 13:40:10.928523 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:40:10.928458    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-9pg7p" event={"ID":"970f3494-f719-471d-818d-04f90719f2a3","Type":"ContainerDied","Data":"41852c03444f993ee800af829b0b67a8e2641ebc8c5acd715132ac15f16c3d7f"}
Jul 02 13:40:10.928523 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:40:10.928498    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41852c03444f993ee800af829b0b67a8e2641ebc8c5acd715132ac15f16c3d7f"
Jul 02 13:40:17.214028 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:40:17.213992    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:40:17.491988 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:40:17.491854    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:40:23.834363 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:40:23.834333    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj"
Jul 02 13:40:29.564661 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:40:29.564625    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:40:29.624100 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:40:29.624075    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:40:31.591651 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:40:31.591615    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:40:48.052994 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:40:48.052955    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:41:19.057146 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:41:19.057095    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:41:19.158616 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:41:19.158577    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:42:08.461581 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:08.461544    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj"]
Jul 02 13:42:08.468753 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:08.468722    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:42:08.471780 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:42:08.471748    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 02 13:42:08.471780 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:42:08.471780 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:42:08.471780 ip-10-0-135-143 kubenswrapper[2583]:  > containerID="20a187fe6a02963b156155bed4ad7336a32eccb851dd30c6e761f115aa7459c1" cmd=["/bin/sleep","15"]
Jul 02 13:42:08.472065 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:08.471805    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj" message="" err=<
Jul 02 13:42:08.472065 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:42:08.472065 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:42:08.472065 ip-10-0-135-143 kubenswrapper[2583]:  >
Jul 02 13:42:08.472065 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:42:08.471822    2583 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 02 13:42:08.472065 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:42:08.472065 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:42:08.472065 ip-10-0-135-143 kubenswrapper[2583]:  > pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj" podUID="9a92ddaf-d371-4a7c-9772-780ca0efe46c" containerName="main" containerID="cri-o://20a187fe6a02963b156155bed4ad7336a32eccb851dd30c6e761f115aa7459c1"
Jul 02 13:42:08.472065 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:08.471868    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj" podUID="9a92ddaf-d371-4a7c-9772-780ca0efe46c" containerName="main" containerID="cri-o://20a187fe6a02963b156155bed4ad7336a32eccb851dd30c6e761f115aa7459c1" gracePeriod=60
Jul 02 13:42:08.732314 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:08.732285    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj"
Jul 02 13:42:08.824740 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:08.824702    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9a92ddaf-d371-4a7c-9772-780ca0efe46c-tokenizer-uds\") pod \"9a92ddaf-d371-4a7c-9772-780ca0efe46c\" (UID: \"9a92ddaf-d371-4a7c-9772-780ca0efe46c\") "
Jul 02 13:42:08.824923 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:08.824755    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a92ddaf-d371-4a7c-9772-780ca0efe46c-tls-certs\") pod \"9a92ddaf-d371-4a7c-9772-780ca0efe46c\" (UID: \"9a92ddaf-d371-4a7c-9772-780ca0efe46c\") "
Jul 02 13:42:08.824923 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:08.824847    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ds8q2\" (UniqueName: \"kubernetes.io/projected/9a92ddaf-d371-4a7c-9772-780ca0efe46c-kube-api-access-ds8q2\") pod \"9a92ddaf-d371-4a7c-9772-780ca0efe46c\" (UID: \"9a92ddaf-d371-4a7c-9772-780ca0efe46c\") "
Jul 02 13:42:08.824923 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:08.824906    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a92ddaf-d371-4a7c-9772-780ca0efe46c-kserve-provision-location\") pod \"9a92ddaf-d371-4a7c-9772-780ca0efe46c\" (UID: \"9a92ddaf-d371-4a7c-9772-780ca0efe46c\") "
Jul 02 13:42:08.825048 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:08.825027    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a92ddaf-d371-4a7c-9772-780ca0efe46c-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "9a92ddaf-d371-4a7c-9772-780ca0efe46c" (UID: "9a92ddaf-d371-4a7c-9772-780ca0efe46c"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:42:08.825228 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:08.825198    2583 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9a92ddaf-d371-4a7c-9772-780ca0efe46c-tokenizer-uds\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:42:08.825657 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:08.825631    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a92ddaf-d371-4a7c-9772-780ca0efe46c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9a92ddaf-d371-4a7c-9772-780ca0efe46c" (UID: "9a92ddaf-d371-4a7c-9772-780ca0efe46c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:42:08.826876 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:08.826844    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a92ddaf-d371-4a7c-9772-780ca0efe46c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9a92ddaf-d371-4a7c-9772-780ca0efe46c" (UID: "9a92ddaf-d371-4a7c-9772-780ca0efe46c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:42:08.826964 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:08.826890    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a92ddaf-d371-4a7c-9772-780ca0efe46c-kube-api-access-ds8q2" (OuterVolumeSpecName: "kube-api-access-ds8q2") pod "9a92ddaf-d371-4a7c-9772-780ca0efe46c" (UID: "9a92ddaf-d371-4a7c-9772-780ca0efe46c"). InnerVolumeSpecName "kube-api-access-ds8q2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:42:08.926142 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:08.926102    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds8q2\" (UniqueName: \"kubernetes.io/projected/9a92ddaf-d371-4a7c-9772-780ca0efe46c-kube-api-access-ds8q2\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:42:08.926142 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:08.926135    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a92ddaf-d371-4a7c-9772-780ca0efe46c-kserve-provision-location\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:42:08.926142 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:08.926145    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a92ddaf-d371-4a7c-9772-780ca0efe46c-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:42:09.516677 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:09.516587    2583 generic.go:356] "Generic (PLEG): container finished" podID="9a92ddaf-d371-4a7c-9772-780ca0efe46c" containerID="20a187fe6a02963b156155bed4ad7336a32eccb851dd30c6e761f115aa7459c1" exitCode=0
Jul 02 13:42:09.516677 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:09.516635    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj"
Jul 02 13:42:09.516677 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:09.516665    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj" event={"ID":"9a92ddaf-d371-4a7c-9772-780ca0efe46c","Type":"ContainerDied","Data":"20a187fe6a02963b156155bed4ad7336a32eccb851dd30c6e761f115aa7459c1"}
Jul 02 13:42:09.517197 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:09.516718    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj" event={"ID":"9a92ddaf-d371-4a7c-9772-780ca0efe46c","Type":"ContainerDied","Data":"628a57f16a793cdec7d5ee586684ddf39524e82b4e68fe62c0a82d96bdb6a934"}
Jul 02 13:42:09.517197 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:09.516744    2583 scope.go:117] "RemoveContainer" containerID="20a187fe6a02963b156155bed4ad7336a32eccb851dd30c6e761f115aa7459c1"
Jul 02 13:42:09.525800 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:09.525776    2583 scope.go:117] "RemoveContainer" containerID="4723cbbed0b9cc0e4de93b41ed40498816a13f2c437e5e5a7fc2b0a0b53753ee"
Jul 02 13:42:09.534457 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:09.534433    2583 scope.go:117] "RemoveContainer" containerID="20a187fe6a02963b156155bed4ad7336a32eccb851dd30c6e761f115aa7459c1"
Jul 02 13:42:09.534823 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:42:09.534797    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20a187fe6a02963b156155bed4ad7336a32eccb851dd30c6e761f115aa7459c1\": container with ID starting with 20a187fe6a02963b156155bed4ad7336a32eccb851dd30c6e761f115aa7459c1 not found: ID does not exist" containerID="20a187fe6a02963b156155bed4ad7336a32eccb851dd30c6e761f115aa7459c1"
Jul 02 13:42:09.534985 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:09.534951    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20a187fe6a02963b156155bed4ad7336a32eccb851dd30c6e761f115aa7459c1"} err="failed to get container status \"20a187fe6a02963b156155bed4ad7336a32eccb851dd30c6e761f115aa7459c1\": rpc error: code = NotFound desc = could not find container \"20a187fe6a02963b156155bed4ad7336a32eccb851dd30c6e761f115aa7459c1\": container with ID starting with 20a187fe6a02963b156155bed4ad7336a32eccb851dd30c6e761f115aa7459c1 not found: ID does not exist"
Jul 02 13:42:09.534985 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:09.534986    2583 scope.go:117] "RemoveContainer" containerID="4723cbbed0b9cc0e4de93b41ed40498816a13f2c437e5e5a7fc2b0a0b53753ee"
Jul 02 13:42:09.535272 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:42:09.535254    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4723cbbed0b9cc0e4de93b41ed40498816a13f2c437e5e5a7fc2b0a0b53753ee\": container with ID starting with 4723cbbed0b9cc0e4de93b41ed40498816a13f2c437e5e5a7fc2b0a0b53753ee not found: ID does not exist" containerID="4723cbbed0b9cc0e4de93b41ed40498816a13f2c437e5e5a7fc2b0a0b53753ee"
Jul 02 13:42:09.535352 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:09.535279    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4723cbbed0b9cc0e4de93b41ed40498816a13f2c437e5e5a7fc2b0a0b53753ee"} err="failed to get container status \"4723cbbed0b9cc0e4de93b41ed40498816a13f2c437e5e5a7fc2b0a0b53753ee\": rpc error: code = NotFound desc = could not find container \"4723cbbed0b9cc0e4de93b41ed40498816a13f2c437e5e5a7fc2b0a0b53753ee\": container with ID starting with 4723cbbed0b9cc0e4de93b41ed40498816a13f2c437e5e5a7fc2b0a0b53753ee not found: ID does not exist"
Jul 02 13:42:09.535352 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:09.535309    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj"]
Jul 02 13:42:09.537063 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:09.537045    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-65477b6fbm6xj"]
Jul 02 13:42:10.031542 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:10.031505    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:42:11.234686 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:11.234653    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a92ddaf-d371-4a7c-9772-780ca0efe46c" path="/var/lib/kubelet/pods/9a92ddaf-d371-4a7c-9772-780ca0efe46c/volumes"
Jul 02 13:42:24.880176 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:24.880141    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:42:35.044082 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:35.044044    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:42:35.679092 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:35.679050    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f"]
Jul 02 13:42:35.679570 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:35.679550    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="970f3494-f719-471d-818d-04f90719f2a3" containerName="storage-initializer"
Jul 02 13:42:35.679570 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:35.679571    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="970f3494-f719-471d-818d-04f90719f2a3" containerName="storage-initializer"
Jul 02 13:42:35.679728 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:35.679588    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a92ddaf-d371-4a7c-9772-780ca0efe46c" containerName="storage-initializer"
Jul 02 13:42:35.679728 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:35.679600    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a92ddaf-d371-4a7c-9772-780ca0efe46c" containerName="storage-initializer"
Jul 02 13:42:35.679728 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:35.679618    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a92ddaf-d371-4a7c-9772-780ca0efe46c" containerName="main"
Jul 02 13:42:35.679728 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:35.679626    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a92ddaf-d371-4a7c-9772-780ca0efe46c" containerName="main"
Jul 02 13:42:35.679728 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:35.679710    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="970f3494-f719-471d-818d-04f90719f2a3" containerName="storage-initializer"
Jul 02 13:42:35.679728 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:35.679728    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a92ddaf-d371-4a7c-9772-780ca0efe46c" containerName="main"
Jul 02 13:42:35.682965 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:35.682941    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f"
Jul 02 13:42:35.685914 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:35.685896    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7s8sh\""
Jul 02 13:42:35.686522 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:35.686503    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-epp-sa-dockercfg-nfbx7\""
Jul 02 13:42:35.686522 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:35.686517    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:42:35.686670 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:35.686506    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 02 13:42:35.693933 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:35.693906    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f"]
Jul 02 13:42:35.726502 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:35.722952    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:42:35.777639 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:35.777609    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdbdde41-c8fd-4024-b3a5-f702dd305fd9-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f\" (UID: \"fdbdde41-c8fd-4024-b3a5-f702dd305fd9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f"
Jul 02 13:42:35.777823 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:35.777671    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jx2w\" (UniqueName: \"kubernetes.io/projected/fdbdde41-c8fd-4024-b3a5-f702dd305fd9-kube-api-access-2jx2w\") pod \"router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f\" (UID: \"fdbdde41-c8fd-4024-b3a5-f702dd305fd9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f"
Jul 02 13:42:35.777823 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:35.777761    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fdbdde41-c8fd-4024-b3a5-f702dd305fd9-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f\" (UID: \"fdbdde41-c8fd-4024-b3a5-f702dd305fd9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f"
Jul 02 13:42:35.777823 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:35.777787    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdbdde41-c8fd-4024-b3a5-f702dd305fd9-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f\" (UID: \"fdbdde41-c8fd-4024-b3a5-f702dd305fd9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f"
Jul 02 13:42:35.777989 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:35.777965    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:42:35.878514 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:35.878465    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jx2w\" (UniqueName: \"kubernetes.io/projected/fdbdde41-c8fd-4024-b3a5-f702dd305fd9-kube-api-access-2jx2w\") pod \"router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f\" (UID: \"fdbdde41-c8fd-4024-b3a5-f702dd305fd9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f"
Jul 02 13:42:35.878723 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:35.878631    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fdbdde41-c8fd-4024-b3a5-f702dd305fd9-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f\" (UID: \"fdbdde41-c8fd-4024-b3a5-f702dd305fd9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f"
Jul 02 13:42:35.878723 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:35.878657    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdbdde41-c8fd-4024-b3a5-f702dd305fd9-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f\" (UID: \"fdbdde41-c8fd-4024-b3a5-f702dd305fd9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f"
Jul 02 13:42:35.878723 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:35.878711    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdbdde41-c8fd-4024-b3a5-f702dd305fd9-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f\" (UID: \"fdbdde41-c8fd-4024-b3a5-f702dd305fd9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f"
Jul 02 13:42:35.879001 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:35.878982    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fdbdde41-c8fd-4024-b3a5-f702dd305fd9-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f\" (UID: \"fdbdde41-c8fd-4024-b3a5-f702dd305fd9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f"
Jul 02 13:42:35.879056 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:35.879036    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdbdde41-c8fd-4024-b3a5-f702dd305fd9-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f\" (UID: \"fdbdde41-c8fd-4024-b3a5-f702dd305fd9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f"
Jul 02 13:42:35.881120 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:35.881102    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdbdde41-c8fd-4024-b3a5-f702dd305fd9-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f\" (UID: \"fdbdde41-c8fd-4024-b3a5-f702dd305fd9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f"
Jul 02 13:42:35.886990 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:35.886967    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jx2w\" (UniqueName: \"kubernetes.io/projected/fdbdde41-c8fd-4024-b3a5-f702dd305fd9-kube-api-access-2jx2w\") pod \"router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f\" (UID: \"fdbdde41-c8fd-4024-b3a5-f702dd305fd9\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f"
Jul 02 13:42:35.993009 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:35.992910    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f"
Jul 02 13:42:36.181224 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:36.181198    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f"]
Jul 02 13:42:36.183777 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:42:36.183749    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdbdde41_c8fd_4024_b3a5_f702dd305fd9.slice/crio-d229e203793e97010db40fc95fb86c4b0ddcdfc9748ff04a09a97ea0f8def0d6 WatchSource:0}: Error finding container d229e203793e97010db40fc95fb86c4b0ddcdfc9748ff04a09a97ea0f8def0d6: Status 404 returned error can't find the container with id d229e203793e97010db40fc95fb86c4b0ddcdfc9748ff04a09a97ea0f8def0d6
Jul 02 13:42:36.658678 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:36.658627    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f" event={"ID":"fdbdde41-c8fd-4024-b3a5-f702dd305fd9","Type":"ContainerStarted","Data":"c9ee3425c3f2cabae14f45dbbbd22692072a221f8ee4a0d0ae7418630335ce45"}
Jul 02 13:42:36.658916 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:36.658744    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f" event={"ID":"fdbdde41-c8fd-4024-b3a5-f702dd305fd9","Type":"ContainerStarted","Data":"d229e203793e97010db40fc95fb86c4b0ddcdfc9748ff04a09a97ea0f8def0d6"}
Jul 02 13:42:37.665867 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:37.665825    2583 generic.go:356] "Generic (PLEG): container finished" podID="fdbdde41-c8fd-4024-b3a5-f702dd305fd9" containerID="c9ee3425c3f2cabae14f45dbbbd22692072a221f8ee4a0d0ae7418630335ce45" exitCode=0
Jul 02 13:42:37.666330 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:37.665918    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f" event={"ID":"fdbdde41-c8fd-4024-b3a5-f702dd305fd9","Type":"ContainerDied","Data":"c9ee3425c3f2cabae14f45dbbbd22692072a221f8ee4a0d0ae7418630335ce45"}
Jul 02 13:42:38.672258 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:38.672215    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f" event={"ID":"fdbdde41-c8fd-4024-b3a5-f702dd305fd9","Type":"ContainerStarted","Data":"ff93027b6747282ed66b0c1ef6fc25cd22d3c48a5860556015fbb82f2c77da7e"}
Jul 02 13:42:38.672709 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:38.672295    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f"
Jul 02 13:42:38.692419 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:38.692352    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f" podStartSLOduration=2.954818475 podStartE2EDuration="3.692335477s" totalImagesPullingTime="0s" totalInitContainerRuntime="737.517002ms" isStatefulPod=false podCreationTimestamp="2026-07-02 13:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:42:38.691473938 +0000 UTC m=+1669.919385111" watchObservedRunningTime="2026-07-02 13:42:38.692335477 +0000 UTC m=+1669.920246651"
Jul 02 13:42:55.984652 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:55.984620    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:42:56.039538 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:42:56.039507    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:43:09.679345 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:43:09.679313    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f"
Jul 02 13:43:26.873758 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:43:26.873663    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:43:27.257308 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:43:27.257178    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:43:39.415063 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:43:39.415016    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:43:39.522856 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:43:39.522818    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:43:39.623469 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:43:39.623423    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:44:46.721696 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:46.721607    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f"]
Jul 02 13:44:46.722742 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:46.722716    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:44:46.731057 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:44:46.731019    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 02 13:44:46.731057 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:44:46.731057 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:44:46.731057 ip-10-0-135-143 kubenswrapper[2583]:  > containerID="ff93027b6747282ed66b0c1ef6fc25cd22d3c48a5860556015fbb82f2c77da7e" cmd=["/bin/sleep","15"]
Jul 02 13:44:46.731381 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:46.731069    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f" message="" err=<
Jul 02 13:44:46.731381 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:44:46.731381 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:44:46.731381 ip-10-0-135-143 kubenswrapper[2583]:  >
Jul 02 13:44:46.731381 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:44:46.731081    2583 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 02 13:44:46.731381 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:44:46.731381 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:44:46.731381 ip-10-0-135-143 kubenswrapper[2583]:  > pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f" podUID="fdbdde41-c8fd-4024-b3a5-f702dd305fd9" containerName="main" containerID="cri-o://ff93027b6747282ed66b0c1ef6fc25cd22d3c48a5860556015fbb82f2c77da7e"
Jul 02 13:44:46.731381 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:46.731120    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f" podUID="fdbdde41-c8fd-4024-b3a5-f702dd305fd9" containerName="main" containerID="cri-o://ff93027b6747282ed66b0c1ef6fc25cd22d3c48a5860556015fbb82f2c77da7e" gracePeriod=60
Jul 02 13:44:46.782233 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:46.782202    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:44:46.992117 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:46.992089    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f"
Jul 02 13:44:47.086280 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:47.086246    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdbdde41-c8fd-4024-b3a5-f702dd305fd9-tls-certs\") pod \"fdbdde41-c8fd-4024-b3a5-f702dd305fd9\" (UID: \"fdbdde41-c8fd-4024-b3a5-f702dd305fd9\") "
Jul 02 13:44:47.086280 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:47.086285    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jx2w\" (UniqueName: \"kubernetes.io/projected/fdbdde41-c8fd-4024-b3a5-f702dd305fd9-kube-api-access-2jx2w\") pod \"fdbdde41-c8fd-4024-b3a5-f702dd305fd9\" (UID: \"fdbdde41-c8fd-4024-b3a5-f702dd305fd9\") "
Jul 02 13:44:47.086551 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:47.086365    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fdbdde41-c8fd-4024-b3a5-f702dd305fd9-tokenizer-uds\") pod \"fdbdde41-c8fd-4024-b3a5-f702dd305fd9\" (UID: \"fdbdde41-c8fd-4024-b3a5-f702dd305fd9\") "
Jul 02 13:44:47.086551 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:47.086405    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdbdde41-c8fd-4024-b3a5-f702dd305fd9-kserve-provision-location\") pod \"fdbdde41-c8fd-4024-b3a5-f702dd305fd9\" (UID: \"fdbdde41-c8fd-4024-b3a5-f702dd305fd9\") "
Jul 02 13:44:47.086667 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:47.086625    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbdde41-c8fd-4024-b3a5-f702dd305fd9-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "fdbdde41-c8fd-4024-b3a5-f702dd305fd9" (UID: "fdbdde41-c8fd-4024-b3a5-f702dd305fd9"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:44:47.086738 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:47.086722    2583 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/fdbdde41-c8fd-4024-b3a5-f702dd305fd9-tokenizer-uds\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:44:47.087177 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:47.087150    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbdde41-c8fd-4024-b3a5-f702dd305fd9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fdbdde41-c8fd-4024-b3a5-f702dd305fd9" (UID: "fdbdde41-c8fd-4024-b3a5-f702dd305fd9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:44:47.088436 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:47.088413    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdbdde41-c8fd-4024-b3a5-f702dd305fd9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fdbdde41-c8fd-4024-b3a5-f702dd305fd9" (UID: "fdbdde41-c8fd-4024-b3a5-f702dd305fd9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:44:47.088492 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:47.088461    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdbdde41-c8fd-4024-b3a5-f702dd305fd9-kube-api-access-2jx2w" (OuterVolumeSpecName: "kube-api-access-2jx2w") pod "fdbdde41-c8fd-4024-b3a5-f702dd305fd9" (UID: "fdbdde41-c8fd-4024-b3a5-f702dd305fd9"). InnerVolumeSpecName "kube-api-access-2jx2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:44:47.191159 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:47.188183    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jx2w\" (UniqueName: \"kubernetes.io/projected/fdbdde41-c8fd-4024-b3a5-f702dd305fd9-kube-api-access-2jx2w\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:44:47.191159 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:47.188220    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdbdde41-c8fd-4024-b3a5-f702dd305fd9-kserve-provision-location\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:44:47.191159 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:47.188236    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdbdde41-c8fd-4024-b3a5-f702dd305fd9-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:44:47.307223 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:47.307189    2583 generic.go:356] "Generic (PLEG): container finished" podID="fdbdde41-c8fd-4024-b3a5-f702dd305fd9" containerID="ff93027b6747282ed66b0c1ef6fc25cd22d3c48a5860556015fbb82f2c77da7e" exitCode=0
Jul 02 13:44:47.307379 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:47.307254    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f"
Jul 02 13:44:47.307379 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:47.307268    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f" event={"ID":"fdbdde41-c8fd-4024-b3a5-f702dd305fd9","Type":"ContainerDied","Data":"ff93027b6747282ed66b0c1ef6fc25cd22d3c48a5860556015fbb82f2c77da7e"}
Jul 02 13:44:47.307379 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:47.307323    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f" event={"ID":"fdbdde41-c8fd-4024-b3a5-f702dd305fd9","Type":"ContainerDied","Data":"d229e203793e97010db40fc95fb86c4b0ddcdfc9748ff04a09a97ea0f8def0d6"}
Jul 02 13:44:47.307379 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:47.307345    2583 scope.go:117] "RemoveContainer" containerID="ff93027b6747282ed66b0c1ef6fc25cd22d3c48a5860556015fbb82f2c77da7e"
Jul 02 13:44:47.316679 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:47.316659    2583 scope.go:117] "RemoveContainer" containerID="c9ee3425c3f2cabae14f45dbbbd22692072a221f8ee4a0d0ae7418630335ce45"
Jul 02 13:44:47.324985 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:47.324959    2583 scope.go:117] "RemoveContainer" containerID="ff93027b6747282ed66b0c1ef6fc25cd22d3c48a5860556015fbb82f2c77da7e"
Jul 02 13:44:47.325276 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:44:47.325255    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff93027b6747282ed66b0c1ef6fc25cd22d3c48a5860556015fbb82f2c77da7e\": container with ID starting with ff93027b6747282ed66b0c1ef6fc25cd22d3c48a5860556015fbb82f2c77da7e not found: ID does not exist" containerID="ff93027b6747282ed66b0c1ef6fc25cd22d3c48a5860556015fbb82f2c77da7e"
Jul 02 13:44:47.325334 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:47.325284    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff93027b6747282ed66b0c1ef6fc25cd22d3c48a5860556015fbb82f2c77da7e"} err="failed to get container status \"ff93027b6747282ed66b0c1ef6fc25cd22d3c48a5860556015fbb82f2c77da7e\": rpc error: code = NotFound desc = could not find container \"ff93027b6747282ed66b0c1ef6fc25cd22d3c48a5860556015fbb82f2c77da7e\": container with ID starting with ff93027b6747282ed66b0c1ef6fc25cd22d3c48a5860556015fbb82f2c77da7e not found: ID does not exist"
Jul 02 13:44:47.325334 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:47.325302    2583 scope.go:117] "RemoveContainer" containerID="c9ee3425c3f2cabae14f45dbbbd22692072a221f8ee4a0d0ae7418630335ce45"
Jul 02 13:44:47.325615 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:44:47.325598    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9ee3425c3f2cabae14f45dbbbd22692072a221f8ee4a0d0ae7418630335ce45\": container with ID starting with c9ee3425c3f2cabae14f45dbbbd22692072a221f8ee4a0d0ae7418630335ce45 not found: ID does not exist" containerID="c9ee3425c3f2cabae14f45dbbbd22692072a221f8ee4a0d0ae7418630335ce45"
Jul 02 13:44:47.325693 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:47.325618    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9ee3425c3f2cabae14f45dbbbd22692072a221f8ee4a0d0ae7418630335ce45"} err="failed to get container status \"c9ee3425c3f2cabae14f45dbbbd22692072a221f8ee4a0d0ae7418630335ce45\": rpc error: code = NotFound desc = could not find container \"c9ee3425c3f2cabae14f45dbbbd22692072a221f8ee4a0d0ae7418630335ce45\": container with ID starting with c9ee3425c3f2cabae14f45dbbbd22692072a221f8ee4a0d0ae7418630335ce45 not found: ID does not exist"
Jul 02 13:44:47.325693 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:47.325605    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f"]
Jul 02 13:44:47.330713 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:47.330689    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7ddbf6dbc7-zp84f"]
Jul 02 13:44:49.233663 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:49.233630    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdbdde41-c8fd-4024-b3a5-f702dd305fd9" path="/var/lib/kubelet/pods/fdbdde41-c8fd-4024-b3a5-f702dd305fd9/volumes"
Jul 02 13:44:49.350409 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:49.350368    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 13:44:49.357333 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:49.357309    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 13:44:49.652935 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:44:49.652898    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:45:16.104333 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:16.104302    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:45:16.351502 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:16.351470    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw"]
Jul 02 13:45:16.351947 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:16.351922    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdbdde41-c8fd-4024-b3a5-f702dd305fd9" containerName="storage-initializer"
Jul 02 13:45:16.351947 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:16.351947    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdbdde41-c8fd-4024-b3a5-f702dd305fd9" containerName="storage-initializer"
Jul 02 13:45:16.352122 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:16.351981    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdbdde41-c8fd-4024-b3a5-f702dd305fd9" containerName="main"
Jul 02 13:45:16.352122 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:16.351986    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdbdde41-c8fd-4024-b3a5-f702dd305fd9" containerName="main"
Jul 02 13:45:16.352122 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:16.352044    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdbdde41-c8fd-4024-b3a5-f702dd305fd9" containerName="main"
Jul 02 13:45:16.355425 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:16.355362    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw"
Jul 02 13:45:16.358193 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:16.358172    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7s8sh\""
Jul 02 13:45:16.358306 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:16.358240    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:45:16.359245 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:16.359230    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-2hsjn\""
Jul 02 13:45:16.359304 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:16.359242    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Jul 02 13:45:16.365166 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:16.365143    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw"]
Jul 02 13:45:16.375321 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:16.375298    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:45:16.447487 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:16.447449    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/81c7fc76-b188-4a22-b9f9-85e249a420f2-tokenizer-uds\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw\" (UID: \"81c7fc76-b188-4a22-b9f9-85e249a420f2\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw"
Jul 02 13:45:16.447656 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:16.447506    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m5hh\" (UniqueName: \"kubernetes.io/projected/81c7fc76-b188-4a22-b9f9-85e249a420f2-kube-api-access-5m5hh\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw\" (UID: \"81c7fc76-b188-4a22-b9f9-85e249a420f2\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw"
Jul 02 13:45:16.447656 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:16.447546    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81c7fc76-b188-4a22-b9f9-85e249a420f2-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw\" (UID: \"81c7fc76-b188-4a22-b9f9-85e249a420f2\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw"
Jul 02 13:45:16.447740 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:16.447654    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81c7fc76-b188-4a22-b9f9-85e249a420f2-kserve-provision-location\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw\" (UID: \"81c7fc76-b188-4a22-b9f9-85e249a420f2\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw"
Jul 02 13:45:16.549052 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:16.549011    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81c7fc76-b188-4a22-b9f9-85e249a420f2-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw\" (UID: \"81c7fc76-b188-4a22-b9f9-85e249a420f2\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw"
Jul 02 13:45:16.549237 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:16.549106    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81c7fc76-b188-4a22-b9f9-85e249a420f2-kserve-provision-location\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw\" (UID: \"81c7fc76-b188-4a22-b9f9-85e249a420f2\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw"
Jul 02 13:45:16.549237 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:16.549158    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/81c7fc76-b188-4a22-b9f9-85e249a420f2-tokenizer-uds\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw\" (UID: \"81c7fc76-b188-4a22-b9f9-85e249a420f2\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw"
Jul 02 13:45:16.549237 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:16.549195    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m5hh\" (UniqueName: \"kubernetes.io/projected/81c7fc76-b188-4a22-b9f9-85e249a420f2-kube-api-access-5m5hh\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw\" (UID: \"81c7fc76-b188-4a22-b9f9-85e249a420f2\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw"
Jul 02 13:45:16.549554 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:16.549526    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81c7fc76-b188-4a22-b9f9-85e249a420f2-kserve-provision-location\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw\" (UID: \"81c7fc76-b188-4a22-b9f9-85e249a420f2\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw"
Jul 02 13:45:16.549637 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:16.549552    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/81c7fc76-b188-4a22-b9f9-85e249a420f2-tokenizer-uds\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw\" (UID: \"81c7fc76-b188-4a22-b9f9-85e249a420f2\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw"
Jul 02 13:45:16.551505 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:16.551481    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81c7fc76-b188-4a22-b9f9-85e249a420f2-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw\" (UID: \"81c7fc76-b188-4a22-b9f9-85e249a420f2\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw"
Jul 02 13:45:16.561359 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:16.561326    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m5hh\" (UniqueName: \"kubernetes.io/projected/81c7fc76-b188-4a22-b9f9-85e249a420f2-kube-api-access-5m5hh\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw\" (UID: \"81c7fc76-b188-4a22-b9f9-85e249a420f2\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw"
Jul 02 13:45:16.665401 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:16.665350    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw"
Jul 02 13:45:16.803168 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:16.803127    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw"]
Jul 02 13:45:16.805511 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:45:16.805485    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81c7fc76_b188_4a22_b9f9_85e249a420f2.slice/crio-42f47e6b4674926b4a8bbe5cea605fffef1d2cff7955507e01b91fe82c89cb7f WatchSource:0}: Error finding container 42f47e6b4674926b4a8bbe5cea605fffef1d2cff7955507e01b91fe82c89cb7f: Status 404 returned error can't find the container with id 42f47e6b4674926b4a8bbe5cea605fffef1d2cff7955507e01b91fe82c89cb7f
Jul 02 13:45:17.460613 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:17.460570    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw" event={"ID":"81c7fc76-b188-4a22-b9f9-85e249a420f2","Type":"ContainerStarted","Data":"ec38d8a46a04b23edfb0a3f73a1219e70a4140c90bcb84b8c2d4d9f657542324"}
Jul 02 13:45:17.460999 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:17.460677    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw" event={"ID":"81c7fc76-b188-4a22-b9f9-85e249a420f2","Type":"ContainerStarted","Data":"42f47e6b4674926b4a8bbe5cea605fffef1d2cff7955507e01b91fe82c89cb7f"}
Jul 02 13:45:18.466426 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:18.466364    2583 generic.go:356] "Generic (PLEG): container finished" podID="81c7fc76-b188-4a22-b9f9-85e249a420f2" containerID="ec38d8a46a04b23edfb0a3f73a1219e70a4140c90bcb84b8c2d4d9f657542324" exitCode=0
Jul 02 13:45:18.466426 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:18.466431    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw" event={"ID":"81c7fc76-b188-4a22-b9f9-85e249a420f2","Type":"ContainerDied","Data":"ec38d8a46a04b23edfb0a3f73a1219e70a4140c90bcb84b8c2d4d9f657542324"}
Jul 02 13:45:19.473375 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:19.473339    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw" event={"ID":"81c7fc76-b188-4a22-b9f9-85e249a420f2","Type":"ContainerStarted","Data":"3f44667865f7ddd07820472c878afe74c6b54382660c572eee39bdee9304e7f6"}
Jul 02 13:45:19.473786 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:19.473528    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw"
Jul 02 13:45:19.494689 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:19.494634    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw" podStartSLOduration=2.7438262140000003 podStartE2EDuration="3.494618111s" totalImagesPullingTime="0s" totalInitContainerRuntime="750.791897ms" isStatefulPod=false podCreationTimestamp="2026-07-02 13:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:45:19.493374945 +0000 UTC m=+1830.721286121" watchObservedRunningTime="2026-07-02 13:45:19.494618111 +0000 UTC m=+1830.722529285"
Jul 02 13:45:50.311495 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:50.311455    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:45:50.480708 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:45:50.480675    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw"
Jul 02 13:46:39.983809 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:46:39.983775    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:46:40.049332 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:46:40.049297    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:48:12.556862 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:12.556825    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:48:24.947517 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:24.947472    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw"]
Jul 02 13:48:24.949353 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:24.949304    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:48:24.957504 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:48:24.957457    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 02 13:48:24.957504 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:48:24.957504 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:48:24.957504 ip-10-0-135-143 kubenswrapper[2583]:  > containerID="3f44667865f7ddd07820472c878afe74c6b54382660c572eee39bdee9304e7f6" cmd=["/bin/sleep","15"]
Jul 02 13:48:24.957764 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:24.957522    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw" message="" err=<
Jul 02 13:48:24.957764 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:48:24.957764 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:48:24.957764 ip-10-0-135-143 kubenswrapper[2583]:  >
Jul 02 13:48:24.957764 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:48:24.957535    2583 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 02 13:48:24.957764 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:48:24.957764 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:48:24.957764 ip-10-0-135-143 kubenswrapper[2583]:  > pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw" podUID="81c7fc76-b188-4a22-b9f9-85e249a420f2" containerName="main" containerID="cri-o://3f44667865f7ddd07820472c878afe74c6b54382660c572eee39bdee9304e7f6"
Jul 02 13:48:24.957764 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:24.957571    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw" podUID="81c7fc76-b188-4a22-b9f9-85e249a420f2" containerName="main" containerID="cri-o://3f44667865f7ddd07820472c878afe74c6b54382660c572eee39bdee9304e7f6" gracePeriod=60
Jul 02 13:48:25.009746 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:25.009708    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:48:25.214153 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:25.214122    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw"
Jul 02 13:48:25.388252 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:25.388222    2583 generic.go:356] "Generic (PLEG): container finished" podID="81c7fc76-b188-4a22-b9f9-85e249a420f2" containerID="3f44667865f7ddd07820472c878afe74c6b54382660c572eee39bdee9304e7f6" exitCode=0
Jul 02 13:48:25.388461 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:25.388270    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw" event={"ID":"81c7fc76-b188-4a22-b9f9-85e249a420f2","Type":"ContainerDied","Data":"3f44667865f7ddd07820472c878afe74c6b54382660c572eee39bdee9304e7f6"}
Jul 02 13:48:25.388461 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:25.388275    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw"
Jul 02 13:48:25.388461 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:25.388310    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw" event={"ID":"81c7fc76-b188-4a22-b9f9-85e249a420f2","Type":"ContainerDied","Data":"42f47e6b4674926b4a8bbe5cea605fffef1d2cff7955507e01b91fe82c89cb7f"}
Jul 02 13:48:25.388461 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:25.388331    2583 scope.go:117] "RemoveContainer" containerID="3f44667865f7ddd07820472c878afe74c6b54382660c572eee39bdee9304e7f6"
Jul 02 13:48:25.395724 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:25.395697    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5m5hh\" (UniqueName: \"kubernetes.io/projected/81c7fc76-b188-4a22-b9f9-85e249a420f2-kube-api-access-5m5hh\") pod \"81c7fc76-b188-4a22-b9f9-85e249a420f2\" (UID: \"81c7fc76-b188-4a22-b9f9-85e249a420f2\") "
Jul 02 13:48:25.395848 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:25.395730    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81c7fc76-b188-4a22-b9f9-85e249a420f2-tls-certs\") pod \"81c7fc76-b188-4a22-b9f9-85e249a420f2\" (UID: \"81c7fc76-b188-4a22-b9f9-85e249a420f2\") "
Jul 02 13:48:25.395848 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:25.395803    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81c7fc76-b188-4a22-b9f9-85e249a420f2-kserve-provision-location\") pod \"81c7fc76-b188-4a22-b9f9-85e249a420f2\" (UID: \"81c7fc76-b188-4a22-b9f9-85e249a420f2\") "
Jul 02 13:48:25.395848 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:25.395830    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/81c7fc76-b188-4a22-b9f9-85e249a420f2-tokenizer-uds\") pod \"81c7fc76-b188-4a22-b9f9-85e249a420f2\" (UID: \"81c7fc76-b188-4a22-b9f9-85e249a420f2\") "
Jul 02 13:48:25.396193 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:25.396146    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81c7fc76-b188-4a22-b9f9-85e249a420f2-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "81c7fc76-b188-4a22-b9f9-85e249a420f2" (UID: "81c7fc76-b188-4a22-b9f9-85e249a420f2"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:48:25.396643 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:25.396613    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81c7fc76-b188-4a22-b9f9-85e249a420f2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "81c7fc76-b188-4a22-b9f9-85e249a420f2" (UID: "81c7fc76-b188-4a22-b9f9-85e249a420f2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:48:25.397651 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:25.397633    2583 scope.go:117] "RemoveContainer" containerID="ec38d8a46a04b23edfb0a3f73a1219e70a4140c90bcb84b8c2d4d9f657542324"
Jul 02 13:48:25.397984 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:25.397952    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81c7fc76-b188-4a22-b9f9-85e249a420f2-kube-api-access-5m5hh" (OuterVolumeSpecName: "kube-api-access-5m5hh") pod "81c7fc76-b188-4a22-b9f9-85e249a420f2" (UID: "81c7fc76-b188-4a22-b9f9-85e249a420f2"). InnerVolumeSpecName "kube-api-access-5m5hh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:48:25.398190 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:25.398173    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81c7fc76-b188-4a22-b9f9-85e249a420f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "81c7fc76-b188-4a22-b9f9-85e249a420f2" (UID: "81c7fc76-b188-4a22-b9f9-85e249a420f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:48:25.415338 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:25.415313    2583 scope.go:117] "RemoveContainer" containerID="3f44667865f7ddd07820472c878afe74c6b54382660c572eee39bdee9304e7f6"
Jul 02 13:48:25.415669 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:48:25.415648    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f44667865f7ddd07820472c878afe74c6b54382660c572eee39bdee9304e7f6\": container with ID starting with 3f44667865f7ddd07820472c878afe74c6b54382660c572eee39bdee9304e7f6 not found: ID does not exist" containerID="3f44667865f7ddd07820472c878afe74c6b54382660c572eee39bdee9304e7f6"
Jul 02 13:48:25.415725 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:25.415678    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f44667865f7ddd07820472c878afe74c6b54382660c572eee39bdee9304e7f6"} err="failed to get container status \"3f44667865f7ddd07820472c878afe74c6b54382660c572eee39bdee9304e7f6\": rpc error: code = NotFound desc = could not find container \"3f44667865f7ddd07820472c878afe74c6b54382660c572eee39bdee9304e7f6\": container with ID starting with 3f44667865f7ddd07820472c878afe74c6b54382660c572eee39bdee9304e7f6 not found: ID does not exist"
Jul 02 13:48:25.415725 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:25.415695    2583 scope.go:117] "RemoveContainer" containerID="ec38d8a46a04b23edfb0a3f73a1219e70a4140c90bcb84b8c2d4d9f657542324"
Jul 02 13:48:25.415986 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:48:25.415966    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec38d8a46a04b23edfb0a3f73a1219e70a4140c90bcb84b8c2d4d9f657542324\": container with ID starting with ec38d8a46a04b23edfb0a3f73a1219e70a4140c90bcb84b8c2d4d9f657542324 not found: ID does not exist" containerID="ec38d8a46a04b23edfb0a3f73a1219e70a4140c90bcb84b8c2d4d9f657542324"
Jul 02 13:48:25.416037 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:25.415991    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec38d8a46a04b23edfb0a3f73a1219e70a4140c90bcb84b8c2d4d9f657542324"} err="failed to get container status \"ec38d8a46a04b23edfb0a3f73a1219e70a4140c90bcb84b8c2d4d9f657542324\": rpc error: code = NotFound desc = could not find container \"ec38d8a46a04b23edfb0a3f73a1219e70a4140c90bcb84b8c2d4d9f657542324\": container with ID starting with ec38d8a46a04b23edfb0a3f73a1219e70a4140c90bcb84b8c2d4d9f657542324 not found: ID does not exist"
Jul 02 13:48:25.496808 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:25.496707    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m5hh\" (UniqueName: \"kubernetes.io/projected/81c7fc76-b188-4a22-b9f9-85e249a420f2-kube-api-access-5m5hh\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:48:25.496808 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:25.496744    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81c7fc76-b188-4a22-b9f9-85e249a420f2-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:48:25.496808 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:25.496757    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81c7fc76-b188-4a22-b9f9-85e249a420f2-kserve-provision-location\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:48:25.496808 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:25.496769    2583 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/81c7fc76-b188-4a22-b9f9-85e249a420f2-tokenizer-uds\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:48:25.712310 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:25.712276    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw"]
Jul 02 13:48:25.716453 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:25.716414    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schemdcdw"]
Jul 02 13:48:27.233351 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:27.233317    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81c7fc76-b188-4a22-b9f9-85e249a420f2" path="/var/lib/kubelet/pods/81c7fc76-b188-4a22-b9f9-85e249a420f2/volumes"
Jul 02 13:48:54.891029 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:54.890970    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:48:55.204882 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:55.204794    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k"]
Jul 02 13:48:55.205281 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:55.205259    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81c7fc76-b188-4a22-b9f9-85e249a420f2" containerName="main"
Jul 02 13:48:55.205355 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:55.205283    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="81c7fc76-b188-4a22-b9f9-85e249a420f2" containerName="main"
Jul 02 13:48:55.205355 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:55.205307    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81c7fc76-b188-4a22-b9f9-85e249a420f2" containerName="storage-initializer"
Jul 02 13:48:55.205355 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:55.205316    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="81c7fc76-b188-4a22-b9f9-85e249a420f2" containerName="storage-initializer"
Jul 02 13:48:55.205549 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:55.205429    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="81c7fc76-b188-4a22-b9f9-85e249a420f2" containerName="main"
Jul 02 13:48:55.208581 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:55.208556    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k"
Jul 02 13:48:55.211796 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:55.211771    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7s8sh\""
Jul 02 13:48:55.211935 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:55.211771    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:48:55.212722 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:55.212700    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 02 13:48:55.212827 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:55.212700    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-2rdjt\""
Jul 02 13:48:55.220558 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:55.220534    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k"]
Jul 02 13:48:55.243043 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:55.243013    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:48:55.263881 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:55.263843    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dfd4836-4ae8-45e9-8848-959de350e9ef-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k\" (UID: \"8dfd4836-4ae8-45e9-8848-959de350e9ef\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k"
Jul 02 13:48:55.263881 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:55.263883    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8dfd4836-4ae8-45e9-8848-959de350e9ef-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k\" (UID: \"8dfd4836-4ae8-45e9-8848-959de350e9ef\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k"
Jul 02 13:48:55.264127 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:55.263992    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8dfd4836-4ae8-45e9-8848-959de350e9ef-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k\" (UID: \"8dfd4836-4ae8-45e9-8848-959de350e9ef\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k"
Jul 02 13:48:55.264127 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:55.264019    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkfdq\" (UniqueName: \"kubernetes.io/projected/8dfd4836-4ae8-45e9-8848-959de350e9ef-kube-api-access-lkfdq\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k\" (UID: \"8dfd4836-4ae8-45e9-8848-959de350e9ef\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k"
Jul 02 13:48:55.364546 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:55.364505    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8dfd4836-4ae8-45e9-8848-959de350e9ef-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k\" (UID: \"8dfd4836-4ae8-45e9-8848-959de350e9ef\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k"
Jul 02 13:48:55.364743 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:55.364551    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkfdq\" (UniqueName: \"kubernetes.io/projected/8dfd4836-4ae8-45e9-8848-959de350e9ef-kube-api-access-lkfdq\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k\" (UID: \"8dfd4836-4ae8-45e9-8848-959de350e9ef\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k"
Jul 02 13:48:55.364915 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:55.364882    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8dfd4836-4ae8-45e9-8848-959de350e9ef-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k\" (UID: \"8dfd4836-4ae8-45e9-8848-959de350e9ef\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k"
Jul 02 13:48:55.364915 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:55.364885    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dfd4836-4ae8-45e9-8848-959de350e9ef-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k\" (UID: \"8dfd4836-4ae8-45e9-8848-959de350e9ef\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k"
Jul 02 13:48:55.365031 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:55.364945    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8dfd4836-4ae8-45e9-8848-959de350e9ef-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k\" (UID: \"8dfd4836-4ae8-45e9-8848-959de350e9ef\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k"
Jul 02 13:48:55.365273 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:55.365249    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8dfd4836-4ae8-45e9-8848-959de350e9ef-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k\" (UID: \"8dfd4836-4ae8-45e9-8848-959de350e9ef\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k"
Jul 02 13:48:55.367238 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:55.367219    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dfd4836-4ae8-45e9-8848-959de350e9ef-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k\" (UID: \"8dfd4836-4ae8-45e9-8848-959de350e9ef\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k"
Jul 02 13:48:55.372782 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:55.372730    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkfdq\" (UniqueName: \"kubernetes.io/projected/8dfd4836-4ae8-45e9-8848-959de350e9ef-kube-api-access-lkfdq\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k\" (UID: \"8dfd4836-4ae8-45e9-8848-959de350e9ef\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k"
Jul 02 13:48:55.519603 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:55.519514    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k"
Jul 02 13:48:55.661023 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:55.660902    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k"]
Jul 02 13:48:55.663795 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:48:55.663766    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dfd4836_4ae8_45e9_8848_959de350e9ef.slice/crio-72b10854d208c7542a56910418449c182ab6b65b5096f9ac95215f5d7e553c96 WatchSource:0}: Error finding container 72b10854d208c7542a56910418449c182ab6b65b5096f9ac95215f5d7e553c96: Status 404 returned error can't find the container with id 72b10854d208c7542a56910418449c182ab6b65b5096f9ac95215f5d7e553c96
Jul 02 13:48:56.542065 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:56.542024    2583 generic.go:356] "Generic (PLEG): container finished" podID="8dfd4836-4ae8-45e9-8848-959de350e9ef" containerID="ce05a8dc9c26273ed129bc9e83786efa0c95c7e3329ede6093f25e400f60d576" exitCode=0
Jul 02 13:48:56.542455 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:56.542110    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k" event={"ID":"8dfd4836-4ae8-45e9-8848-959de350e9ef","Type":"ContainerDied","Data":"ce05a8dc9c26273ed129bc9e83786efa0c95c7e3329ede6093f25e400f60d576"}
Jul 02 13:48:56.542455 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:56.542164    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k" event={"ID":"8dfd4836-4ae8-45e9-8848-959de350e9ef","Type":"ContainerStarted","Data":"72b10854d208c7542a56910418449c182ab6b65b5096f9ac95215f5d7e553c96"}
Jul 02 13:48:57.548980 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:57.548927    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k" event={"ID":"8dfd4836-4ae8-45e9-8848-959de350e9ef","Type":"ContainerStarted","Data":"8a76f11ca85bc17c39d1e8e6dae1dad45d19da8ecf6a6a2d7d9b5ebac942574d"}
Jul 02 13:48:57.549470 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:57.548992    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k"
Jul 02 13:48:57.570581 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:48:57.570521    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k" podStartSLOduration=1.837759326 podStartE2EDuration="2.570503759s" totalImagesPullingTime="0s" totalInitContainerRuntime="732.744433ms" isStatefulPod=false podCreationTimestamp="2026-07-02 13:48:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:48:57.567809656 +0000 UTC m=+2048.795720829" watchObservedRunningTime="2026-07-02 13:48:57.570503759 +0000 UTC m=+2048.798414932"
Jul 02 13:49:28.555926 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:49:28.555843    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k"
Jul 02 13:49:32.174735 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:49:32.174697    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:49:35.405864 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:49:35.405832    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:49:35.506659 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:49:35.506626    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:49:49.386009 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:49:49.385979    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 13:49:49.394633 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:49:49.394609    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 13:52:22.552592 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:22.552489    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k"]
Jul 02 13:52:22.562946 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:52:22.562907    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 02 13:52:22.562946 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:52:22.562946 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:52:22.562946 ip-10-0-135-143 kubenswrapper[2583]:  > containerID="8a76f11ca85bc17c39d1e8e6dae1dad45d19da8ecf6a6a2d7d9b5ebac942574d" cmd=["/bin/sleep","15"]
Jul 02 13:52:22.563227 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:22.562959    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k" message="" err=<
Jul 02 13:52:22.563227 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:52:22.563227 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:52:22.563227 ip-10-0-135-143 kubenswrapper[2583]:  >
Jul 02 13:52:22.563227 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:52:22.562972    2583 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 02 13:52:22.563227 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:52:22.563227 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:52:22.563227 ip-10-0-135-143 kubenswrapper[2583]:  > pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k" podUID="8dfd4836-4ae8-45e9-8848-959de350e9ef" containerName="main" containerID="cri-o://8a76f11ca85bc17c39d1e8e6dae1dad45d19da8ecf6a6a2d7d9b5ebac942574d"
Jul 02 13:52:22.563227 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:22.563022    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k" podUID="8dfd4836-4ae8-45e9-8848-959de350e9ef" containerName="main" containerID="cri-o://8a76f11ca85bc17c39d1e8e6dae1dad45d19da8ecf6a6a2d7d9b5ebac942574d" gracePeriod=60
Jul 02 13:52:22.594205 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:22.594167    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:52:22.678325 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:22.678288    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:52:22.819715 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:22.819689    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k"
Jul 02 13:52:22.933781 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:22.933745    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dfd4836-4ae8-45e9-8848-959de350e9ef-tls-certs\") pod \"8dfd4836-4ae8-45e9-8848-959de350e9ef\" (UID: \"8dfd4836-4ae8-45e9-8848-959de350e9ef\") "
Jul 02 13:52:22.933993 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:22.933824    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkfdq\" (UniqueName: \"kubernetes.io/projected/8dfd4836-4ae8-45e9-8848-959de350e9ef-kube-api-access-lkfdq\") pod \"8dfd4836-4ae8-45e9-8848-959de350e9ef\" (UID: \"8dfd4836-4ae8-45e9-8848-959de350e9ef\") "
Jul 02 13:52:22.933993 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:22.933890    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8dfd4836-4ae8-45e9-8848-959de350e9ef-tokenizer-uds\") pod \"8dfd4836-4ae8-45e9-8848-959de350e9ef\" (UID: \"8dfd4836-4ae8-45e9-8848-959de350e9ef\") "
Jul 02 13:52:22.933993 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:22.933927    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8dfd4836-4ae8-45e9-8848-959de350e9ef-kserve-provision-location\") pod \"8dfd4836-4ae8-45e9-8848-959de350e9ef\" (UID: \"8dfd4836-4ae8-45e9-8848-959de350e9ef\") "
Jul 02 13:52:22.934181 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:22.934145    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dfd4836-4ae8-45e9-8848-959de350e9ef-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "8dfd4836-4ae8-45e9-8848-959de350e9ef" (UID: "8dfd4836-4ae8-45e9-8848-959de350e9ef"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:52:22.934266 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:22.934250    2583 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8dfd4836-4ae8-45e9-8848-959de350e9ef-tokenizer-uds\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:52:22.934639 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:22.934616    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dfd4836-4ae8-45e9-8848-959de350e9ef-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8dfd4836-4ae8-45e9-8848-959de350e9ef" (UID: "8dfd4836-4ae8-45e9-8848-959de350e9ef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:52:22.935880 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:22.935859    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dfd4836-4ae8-45e9-8848-959de350e9ef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8dfd4836-4ae8-45e9-8848-959de350e9ef" (UID: "8dfd4836-4ae8-45e9-8848-959de350e9ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:52:22.935961 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:22.935948    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dfd4836-4ae8-45e9-8848-959de350e9ef-kube-api-access-lkfdq" (OuterVolumeSpecName: "kube-api-access-lkfdq") pod "8dfd4836-4ae8-45e9-8848-959de350e9ef" (UID: "8dfd4836-4ae8-45e9-8848-959de350e9ef"). InnerVolumeSpecName "kube-api-access-lkfdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:52:23.034729 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:23.034690    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dfd4836-4ae8-45e9-8848-959de350e9ef-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:52:23.034729 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:23.034721    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkfdq\" (UniqueName: \"kubernetes.io/projected/8dfd4836-4ae8-45e9-8848-959de350e9ef-kube-api-access-lkfdq\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:52:23.034729 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:23.034735    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8dfd4836-4ae8-45e9-8848-959de350e9ef-kserve-provision-location\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:52:23.553346 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:23.553249    2583 generic.go:356] "Generic (PLEG): container finished" podID="8dfd4836-4ae8-45e9-8848-959de350e9ef" containerID="8a76f11ca85bc17c39d1e8e6dae1dad45d19da8ecf6a6a2d7d9b5ebac942574d" exitCode=0
Jul 02 13:52:23.553346 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:23.553313    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k" event={"ID":"8dfd4836-4ae8-45e9-8848-959de350e9ef","Type":"ContainerDied","Data":"8a76f11ca85bc17c39d1e8e6dae1dad45d19da8ecf6a6a2d7d9b5ebac942574d"}
Jul 02 13:52:23.553346 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:23.553318    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k"
Jul 02 13:52:23.553943 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:23.553379    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k" event={"ID":"8dfd4836-4ae8-45e9-8848-959de350e9ef","Type":"ContainerDied","Data":"72b10854d208c7542a56910418449c182ab6b65b5096f9ac95215f5d7e553c96"}
Jul 02 13:52:23.553943 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:23.553425    2583 scope.go:117] "RemoveContainer" containerID="8a76f11ca85bc17c39d1e8e6dae1dad45d19da8ecf6a6a2d7d9b5ebac942574d"
Jul 02 13:52:23.562819 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:23.562794    2583 scope.go:117] "RemoveContainer" containerID="ce05a8dc9c26273ed129bc9e83786efa0c95c7e3329ede6093f25e400f60d576"
Jul 02 13:52:23.572081 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:23.572056    2583 scope.go:117] "RemoveContainer" containerID="8a76f11ca85bc17c39d1e8e6dae1dad45d19da8ecf6a6a2d7d9b5ebac942574d"
Jul 02 13:52:23.572184 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:23.572165    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k"]
Jul 02 13:52:23.572448 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:52:23.572426    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a76f11ca85bc17c39d1e8e6dae1dad45d19da8ecf6a6a2d7d9b5ebac942574d\": container with ID starting with 8a76f11ca85bc17c39d1e8e6dae1dad45d19da8ecf6a6a2d7d9b5ebac942574d not found: ID does not exist" containerID="8a76f11ca85bc17c39d1e8e6dae1dad45d19da8ecf6a6a2d7d9b5ebac942574d"
Jul 02 13:52:23.572498 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:23.572458    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a76f11ca85bc17c39d1e8e6dae1dad45d19da8ecf6a6a2d7d9b5ebac942574d"} err="failed to get container status \"8a76f11ca85bc17c39d1e8e6dae1dad45d19da8ecf6a6a2d7d9b5ebac942574d\": rpc error: code = NotFound desc = could not find container \"8a76f11ca85bc17c39d1e8e6dae1dad45d19da8ecf6a6a2d7d9b5ebac942574d\": container with ID starting with 8a76f11ca85bc17c39d1e8e6dae1dad45d19da8ecf6a6a2d7d9b5ebac942574d not found: ID does not exist"
Jul 02 13:52:23.572498 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:23.572477    2583 scope.go:117] "RemoveContainer" containerID="ce05a8dc9c26273ed129bc9e83786efa0c95c7e3329ede6093f25e400f60d576"
Jul 02 13:52:23.572763 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:52:23.572744    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce05a8dc9c26273ed129bc9e83786efa0c95c7e3329ede6093f25e400f60d576\": container with ID starting with ce05a8dc9c26273ed129bc9e83786efa0c95c7e3329ede6093f25e400f60d576 not found: ID does not exist" containerID="ce05a8dc9c26273ed129bc9e83786efa0c95c7e3329ede6093f25e400f60d576"
Jul 02 13:52:23.572807 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:23.572770    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce05a8dc9c26273ed129bc9e83786efa0c95c7e3329ede6093f25e400f60d576"} err="failed to get container status \"ce05a8dc9c26273ed129bc9e83786efa0c95c7e3329ede6093f25e400f60d576\": rpc error: code = NotFound desc = could not find container \"ce05a8dc9c26273ed129bc9e83786efa0c95c7e3329ede6093f25e400f60d576\": container with ID starting with ce05a8dc9c26273ed129bc9e83786efa0c95c7e3329ede6093f25e400f60d576 not found: ID does not exist"
Jul 02 13:52:23.576078 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:23.576053    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-69fd92qc9k"]
Jul 02 13:52:25.234251 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:25.234212    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dfd4836-4ae8-45e9-8848-959de350e9ef" path="/var/lib/kubelet/pods/8dfd4836-4ae8-45e9-8848-959de350e9ef/volumes"
Jul 02 13:52:41.829001 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:41.828960    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"]
Jul 02 13:52:41.829623 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:41.829602    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dfd4836-4ae8-45e9-8848-959de350e9ef" containerName="main"
Jul 02 13:52:41.829676 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:41.829627    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dfd4836-4ae8-45e9-8848-959de350e9ef" containerName="main"
Jul 02 13:52:41.829676 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:41.829659    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dfd4836-4ae8-45e9-8848-959de350e9ef" containerName="storage-initializer"
Jul 02 13:52:41.829676 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:41.829669    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dfd4836-4ae8-45e9-8848-959de350e9ef" containerName="storage-initializer"
Jul 02 13:52:41.829790 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:41.829772    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dfd4836-4ae8-45e9-8848-959de350e9ef" containerName="main"
Jul 02 13:52:41.833075 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:41.833054    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:41.836377 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:41.836352    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-2-openshift-default-dockercfg-wzvxs\""
Jul 02 13:52:41.847512 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:41.847488    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"]
Jul 02 13:52:41.861603 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:41.861576    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:52:41.904370 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:41.904333    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1875f4df-a74b-4235-8bfd-c2e6b8aa64aa-istio-podinfo\") pod \"router-gateway-2-openshift-default-684dc9b4b9-snkw7\" (UID: \"1875f4df-a74b-4235-8bfd-c2e6b8aa64aa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:41.904581 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:41.904412    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1875f4df-a74b-4235-8bfd-c2e6b8aa64aa-istio-envoy\") pod \"router-gateway-2-openshift-default-684dc9b4b9-snkw7\" (UID: \"1875f4df-a74b-4235-8bfd-c2e6b8aa64aa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:41.904581 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:41.904435    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1875f4df-a74b-4235-8bfd-c2e6b8aa64aa-istio-data\") pod \"router-gateway-2-openshift-default-684dc9b4b9-snkw7\" (UID: \"1875f4df-a74b-4235-8bfd-c2e6b8aa64aa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:41.904581 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:41.904479    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1875f4df-a74b-4235-8bfd-c2e6b8aa64aa-credential-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-snkw7\" (UID: \"1875f4df-a74b-4235-8bfd-c2e6b8aa64aa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:41.904581 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:41.904514    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1875f4df-a74b-4235-8bfd-c2e6b8aa64aa-workload-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-snkw7\" (UID: \"1875f4df-a74b-4235-8bfd-c2e6b8aa64aa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:41.904717 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:41.904586    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1875f4df-a74b-4235-8bfd-c2e6b8aa64aa-workload-certs\") pod \"router-gateway-2-openshift-default-684dc9b4b9-snkw7\" (UID: \"1875f4df-a74b-4235-8bfd-c2e6b8aa64aa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:41.904717 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:41.904645    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1875f4df-a74b-4235-8bfd-c2e6b8aa64aa-istio-token\") pod \"router-gateway-2-openshift-default-684dc9b4b9-snkw7\" (UID: \"1875f4df-a74b-4235-8bfd-c2e6b8aa64aa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:41.904717 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:41.904670    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1875f4df-a74b-4235-8bfd-c2e6b8aa64aa-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-684dc9b4b9-snkw7\" (UID: \"1875f4df-a74b-4235-8bfd-c2e6b8aa64aa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:41.904717 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:41.904685    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppq28\" (UniqueName: \"kubernetes.io/projected/1875f4df-a74b-4235-8bfd-c2e6b8aa64aa-kube-api-access-ppq28\") pod \"router-gateway-2-openshift-default-684dc9b4b9-snkw7\" (UID: \"1875f4df-a74b-4235-8bfd-c2e6b8aa64aa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:41.962163 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:41.962130    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:52:42.005896 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:42.005859    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1875f4df-a74b-4235-8bfd-c2e6b8aa64aa-istio-token\") pod \"router-gateway-2-openshift-default-684dc9b4b9-snkw7\" (UID: \"1875f4df-a74b-4235-8bfd-c2e6b8aa64aa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:42.005896 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:42.005901    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1875f4df-a74b-4235-8bfd-c2e6b8aa64aa-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-684dc9b4b9-snkw7\" (UID: \"1875f4df-a74b-4235-8bfd-c2e6b8aa64aa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:42.006134 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:42.005919    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppq28\" (UniqueName: \"kubernetes.io/projected/1875f4df-a74b-4235-8bfd-c2e6b8aa64aa-kube-api-access-ppq28\") pod \"router-gateway-2-openshift-default-684dc9b4b9-snkw7\" (UID: \"1875f4df-a74b-4235-8bfd-c2e6b8aa64aa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:42.006134 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:42.006116    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1875f4df-a74b-4235-8bfd-c2e6b8aa64aa-istio-podinfo\") pod \"router-gateway-2-openshift-default-684dc9b4b9-snkw7\" (UID: \"1875f4df-a74b-4235-8bfd-c2e6b8aa64aa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:42.006213 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:42.006199    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1875f4df-a74b-4235-8bfd-c2e6b8aa64aa-istio-envoy\") pod \"router-gateway-2-openshift-default-684dc9b4b9-snkw7\" (UID: \"1875f4df-a74b-4235-8bfd-c2e6b8aa64aa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:42.006307 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:42.006227    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1875f4df-a74b-4235-8bfd-c2e6b8aa64aa-istio-data\") pod \"router-gateway-2-openshift-default-684dc9b4b9-snkw7\" (UID: \"1875f4df-a74b-4235-8bfd-c2e6b8aa64aa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:42.006307 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:42.006287    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1875f4df-a74b-4235-8bfd-c2e6b8aa64aa-credential-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-snkw7\" (UID: \"1875f4df-a74b-4235-8bfd-c2e6b8aa64aa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:42.006418 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:42.006325    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1875f4df-a74b-4235-8bfd-c2e6b8aa64aa-workload-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-snkw7\" (UID: \"1875f4df-a74b-4235-8bfd-c2e6b8aa64aa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:42.006418 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:42.006371    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1875f4df-a74b-4235-8bfd-c2e6b8aa64aa-workload-certs\") pod \"router-gateway-2-openshift-default-684dc9b4b9-snkw7\" (UID: \"1875f4df-a74b-4235-8bfd-c2e6b8aa64aa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:42.006708 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:42.006677    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1875f4df-a74b-4235-8bfd-c2e6b8aa64aa-credential-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-snkw7\" (UID: \"1875f4df-a74b-4235-8bfd-c2e6b8aa64aa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:42.006851 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:42.006719    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1875f4df-a74b-4235-8bfd-c2e6b8aa64aa-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-684dc9b4b9-snkw7\" (UID: \"1875f4df-a74b-4235-8bfd-c2e6b8aa64aa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:42.006851 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:42.006732    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1875f4df-a74b-4235-8bfd-c2e6b8aa64aa-istio-data\") pod \"router-gateway-2-openshift-default-684dc9b4b9-snkw7\" (UID: \"1875f4df-a74b-4235-8bfd-c2e6b8aa64aa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:42.006851 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:42.006783    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1875f4df-a74b-4235-8bfd-c2e6b8aa64aa-workload-certs\") pod \"router-gateway-2-openshift-default-684dc9b4b9-snkw7\" (UID: \"1875f4df-a74b-4235-8bfd-c2e6b8aa64aa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:42.006851 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:42.006823    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1875f4df-a74b-4235-8bfd-c2e6b8aa64aa-workload-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-snkw7\" (UID: \"1875f4df-a74b-4235-8bfd-c2e6b8aa64aa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:42.008484 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:42.008463    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1875f4df-a74b-4235-8bfd-c2e6b8aa64aa-istio-envoy\") pod \"router-gateway-2-openshift-default-684dc9b4b9-snkw7\" (UID: \"1875f4df-a74b-4235-8bfd-c2e6b8aa64aa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:42.008632 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:42.008615    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1875f4df-a74b-4235-8bfd-c2e6b8aa64aa-istio-podinfo\") pod \"router-gateway-2-openshift-default-684dc9b4b9-snkw7\" (UID: \"1875f4df-a74b-4235-8bfd-c2e6b8aa64aa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:42.014180 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:42.014155    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppq28\" (UniqueName: \"kubernetes.io/projected/1875f4df-a74b-4235-8bfd-c2e6b8aa64aa-kube-api-access-ppq28\") pod \"router-gateway-2-openshift-default-684dc9b4b9-snkw7\" (UID: \"1875f4df-a74b-4235-8bfd-c2e6b8aa64aa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:42.014480 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:42.014464    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1875f4df-a74b-4235-8bfd-c2e6b8aa64aa-istio-token\") pod \"router-gateway-2-openshift-default-684dc9b4b9-snkw7\" (UID: \"1875f4df-a74b-4235-8bfd-c2e6b8aa64aa\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:42.143601 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:42.143566    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:42.281863 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:42.281813    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"]
Jul 02 13:52:42.284095 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:52:42.284065    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1875f4df_a74b_4235_8bfd_c2e6b8aa64aa.slice/crio-aeddbeec3866261f6ed78021bd637de9ef38c039f89bf8e8117e1133adc3c12e WatchSource:0}: Error finding container aeddbeec3866261f6ed78021bd637de9ef38c039f89bf8e8117e1133adc3c12e: Status 404 returned error can't find the container with id aeddbeec3866261f6ed78021bd637de9ef38c039f89bf8e8117e1133adc3c12e
Jul 02 13:52:42.286201 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:42.286169    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 02 13:52:42.286297 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:42.286249    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 02 13:52:42.286297 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:42.286290    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 02 13:52:42.648737 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:42.648694    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7" event={"ID":"1875f4df-a74b-4235-8bfd-c2e6b8aa64aa","Type":"ContainerStarted","Data":"29ffc015d6e13a690e8bb99557ce68d2b1523db3a069cb0a876dc2c37915dabe"}
Jul 02 13:52:42.648936 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:42.648758    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7" event={"ID":"1875f4df-a74b-4235-8bfd-c2e6b8aa64aa","Type":"ContainerStarted","Data":"aeddbeec3866261f6ed78021bd637de9ef38c039f89bf8e8117e1133adc3c12e"}
Jul 02 13:52:42.668090 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:42.668028    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7" podStartSLOduration=1.668008592 podStartE2EDuration="1.668008592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:52:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:52:42.667096021 +0000 UTC m=+2273.895007197" watchObservedRunningTime="2026-07-02 13:52:42.668008592 +0000 UTC m=+2273.895919769"
Jul 02 13:52:43.144490 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:43.144454    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:43.149504 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:43.149480    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:43.653917 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:43.653879    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:43.654964 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:43.654946    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:46.041076 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:46.041043    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:52:56.717819 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:56.717784    2583 generic.go:356] "Generic (PLEG): container finished" podID="1875f4df-a74b-4235-8bfd-c2e6b8aa64aa" containerID="29ffc015d6e13a690e8bb99557ce68d2b1523db3a069cb0a876dc2c37915dabe" exitCode=0
Jul 02 13:52:56.718247 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:56.717856    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7" event={"ID":"1875f4df-a74b-4235-8bfd-c2e6b8aa64aa","Type":"ContainerDied","Data":"29ffc015d6e13a690e8bb99557ce68d2b1523db3a069cb0a876dc2c37915dabe"}
Jul 02 13:52:56.718247 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:56.718238    2583 scope.go:117] "RemoveContainer" containerID="29ffc015d6e13a690e8bb99557ce68d2b1523db3a069cb0a876dc2c37915dabe"
Jul 02 13:52:56.719247 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:56.719220    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 02 13:52:56.719313 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:56.719284    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 02 13:52:56.719353 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:56.719319    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 02 13:52:57.727923 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:57.727887    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7" event={"ID":"1875f4df-a74b-4235-8bfd-c2e6b8aa64aa","Type":"ContainerStarted","Data":"a0ce36f1a6e9899e6109ca58007605e60159607366303dab502fe8ba6c35307d"}
Jul 02 13:52:58.143917 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:58.143880    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:58.145101 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:58.145073    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7" podUID="1875f4df-a74b-4235-8bfd-c2e6b8aa64aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Jul 02 13:52:58.654516 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:58.654462    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:52:59.144600 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:52:59.144554    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7" podUID="1875f4df-a74b-4235-8bfd-c2e6b8aa64aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Jul 02 13:53:00.144935 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:00.144892    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7" podUID="1875f4df-a74b-4235-8bfd-c2e6b8aa64aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Jul 02 13:53:01.144754 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:01.144709    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7" podUID="1875f4df-a74b-4235-8bfd-c2e6b8aa64aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Jul 02 13:53:02.144048 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:02.144002    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7" podUID="1875f4df-a74b-4235-8bfd-c2e6b8aa64aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Jul 02 13:53:03.148550 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:03.148517    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:53:03.149660 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:03.149640    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-snkw7"
Jul 02 13:53:03.329027 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:03.328992    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:53:03.975034 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:03.974998    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj"]
Jul 02 13:53:03.977670 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:03.977647    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj"
Jul 02 13:53:03.980070 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:03.980047    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:53:03.980199 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:03.980165    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7s8sh\""
Jul 02 13:53:03.981179 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:03.981157    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 02 13:53:03.981247 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:03.981201    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-epp-sa-dockercfg-phsgt\""
Jul 02 13:53:03.989719 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:03.989690    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj"]
Jul 02 13:53:03.997894 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:03.997867    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:53:04.079679 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:04.079615    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:53:04.120106 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:04.120075    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2b91412d-7966-4d88-a431-ad3ac9b3041b-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj\" (UID: \"2b91412d-7966-4d88-a431-ad3ac9b3041b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj"
Jul 02 13:53:04.120276 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:04.120115    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b91412d-7966-4d88-a431-ad3ac9b3041b-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj\" (UID: \"2b91412d-7966-4d88-a431-ad3ac9b3041b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj"
Jul 02 13:53:04.120276 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:04.120225    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b91412d-7966-4d88-a431-ad3ac9b3041b-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj\" (UID: \"2b91412d-7966-4d88-a431-ad3ac9b3041b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj"
Jul 02 13:53:04.120355 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:04.120284    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdg8g\" (UniqueName: \"kubernetes.io/projected/2b91412d-7966-4d88-a431-ad3ac9b3041b-kube-api-access-sdg8g\") pod \"router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj\" (UID: \"2b91412d-7966-4d88-a431-ad3ac9b3041b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj"
Jul 02 13:53:04.221587 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:04.221538    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2b91412d-7966-4d88-a431-ad3ac9b3041b-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj\" (UID: \"2b91412d-7966-4d88-a431-ad3ac9b3041b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj"
Jul 02 13:53:04.221587 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:04.221597    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b91412d-7966-4d88-a431-ad3ac9b3041b-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj\" (UID: \"2b91412d-7966-4d88-a431-ad3ac9b3041b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj"
Jul 02 13:53:04.222087 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:04.221641    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b91412d-7966-4d88-a431-ad3ac9b3041b-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj\" (UID: \"2b91412d-7966-4d88-a431-ad3ac9b3041b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj"
Jul 02 13:53:04.222087 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:04.221699    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdg8g\" (UniqueName: \"kubernetes.io/projected/2b91412d-7966-4d88-a431-ad3ac9b3041b-kube-api-access-sdg8g\") pod \"router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj\" (UID: \"2b91412d-7966-4d88-a431-ad3ac9b3041b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj"
Jul 02 13:53:04.222087 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:04.221919    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2b91412d-7966-4d88-a431-ad3ac9b3041b-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj\" (UID: \"2b91412d-7966-4d88-a431-ad3ac9b3041b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj"
Jul 02 13:53:04.222087 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:04.222006    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b91412d-7966-4d88-a431-ad3ac9b3041b-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj\" (UID: \"2b91412d-7966-4d88-a431-ad3ac9b3041b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj"
Jul 02 13:53:04.224188 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:04.224161    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b91412d-7966-4d88-a431-ad3ac9b3041b-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj\" (UID: \"2b91412d-7966-4d88-a431-ad3ac9b3041b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj"
Jul 02 13:53:04.230162 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:04.230098    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdg8g\" (UniqueName: \"kubernetes.io/projected/2b91412d-7966-4d88-a431-ad3ac9b3041b-kube-api-access-sdg8g\") pod \"router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj\" (UID: \"2b91412d-7966-4d88-a431-ad3ac9b3041b\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj"
Jul 02 13:53:04.286983 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:04.286938    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj"
Jul 02 13:53:04.426368 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:04.426342    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj"]
Jul 02 13:53:04.429217 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:53:04.429190    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b91412d_7966_4d88_a431_ad3ac9b3041b.slice/crio-9a2efa1afbb31d6cf1e59f35cd34e3fda0413e31c90082ed9825d8c90bd161d4 WatchSource:0}: Error finding container 9a2efa1afbb31d6cf1e59f35cd34e3fda0413e31c90082ed9825d8c90bd161d4: Status 404 returned error can't find the container with id 9a2efa1afbb31d6cf1e59f35cd34e3fda0413e31c90082ed9825d8c90bd161d4
Jul 02 13:53:04.764880 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:04.764787    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj" event={"ID":"2b91412d-7966-4d88-a431-ad3ac9b3041b","Type":"ContainerStarted","Data":"2154243c7a857bb93a06f866fa8bd36910b24cebc7410f2a4ba88daeafd2016f"}
Jul 02 13:53:04.765042 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:04.764900    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj" event={"ID":"2b91412d-7966-4d88-a431-ad3ac9b3041b","Type":"ContainerStarted","Data":"9a2efa1afbb31d6cf1e59f35cd34e3fda0413e31c90082ed9825d8c90bd161d4"}
Jul 02 13:53:05.770744 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:05.770710    2583 generic.go:356] "Generic (PLEG): container finished" podID="2b91412d-7966-4d88-a431-ad3ac9b3041b" containerID="2154243c7a857bb93a06f866fa8bd36910b24cebc7410f2a4ba88daeafd2016f" exitCode=0
Jul 02 13:53:05.771180 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:05.770801    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj" event={"ID":"2b91412d-7966-4d88-a431-ad3ac9b3041b","Type":"ContainerDied","Data":"2154243c7a857bb93a06f866fa8bd36910b24cebc7410f2a4ba88daeafd2016f"}
Jul 02 13:53:06.777426 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:06.777370    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj" event={"ID":"2b91412d-7966-4d88-a431-ad3ac9b3041b","Type":"ContainerStarted","Data":"9c7f83b0bb3654c567f5ca2d84299a00842c3d7a7d0c67b55b0976b135e7f322"}
Jul 02 13:53:06.777878 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:06.777510    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj"
Jul 02 13:53:06.799637 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:06.799589    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj" podStartSLOduration=3.140769534 podStartE2EDuration="3.799573691s" totalImagesPullingTime="0s" totalInitContainerRuntime="658.804157ms" isStatefulPod=false podCreationTimestamp="2026-07-02 13:53:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:53:06.797331794 +0000 UTC m=+2298.025242969" watchObservedRunningTime="2026-07-02 13:53:06.799573691 +0000 UTC m=+2298.027484865"
Jul 02 13:53:37.784649 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:37.784618    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj"
Jul 02 13:53:43.627115 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:53:43.627028    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:54:49.419882 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:54:49.419850    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 13:54:49.428569 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:54:49.428543    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 13:56:03.509705 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:56:03.509667    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj"]
Jul 02 13:56:03.520991 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:56:03.520945    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 02 13:56:03.520991 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:56:03.520991 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:56:03.520991 ip-10-0-135-143 kubenswrapper[2583]:  > containerID="9c7f83b0bb3654c567f5ca2d84299a00842c3d7a7d0c67b55b0976b135e7f322" cmd=["/bin/sleep","15"]
Jul 02 13:56:03.521341 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:56:03.521007    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj" message="" err=<
Jul 02 13:56:03.521341 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:56:03.521341 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:56:03.521341 ip-10-0-135-143 kubenswrapper[2583]:  >
Jul 02 13:56:03.521341 ip-10-0-135-143 kubenswrapper[2583]: E0702 13:56:03.521020    2583 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 02 13:56:03.521341 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 13:56:03.521341 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 13:56:03.521341 ip-10-0-135-143 kubenswrapper[2583]:  > pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj" podUID="2b91412d-7966-4d88-a431-ad3ac9b3041b" containerName="main" containerID="cri-o://9c7f83b0bb3654c567f5ca2d84299a00842c3d7a7d0c67b55b0976b135e7f322"
Jul 02 13:56:03.521341 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:56:03.521075    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj" podUID="2b91412d-7966-4d88-a431-ad3ac9b3041b" containerName="main" containerID="cri-o://9c7f83b0bb3654c567f5ca2d84299a00842c3d7a7d0c67b55b0976b135e7f322" gracePeriod=60
Jul 02 13:56:03.564336 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:56:03.564284    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:56:03.652439 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:56:03.652404    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:56:03.662229 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:56:03.662196    2583 generic.go:356] "Generic (PLEG): container finished" podID="2b91412d-7966-4d88-a431-ad3ac9b3041b" containerID="9c7f83b0bb3654c567f5ca2d84299a00842c3d7a7d0c67b55b0976b135e7f322" exitCode=0
Jul 02 13:56:03.662419 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:56:03.662253    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj" event={"ID":"2b91412d-7966-4d88-a431-ad3ac9b3041b","Type":"ContainerDied","Data":"9c7f83b0bb3654c567f5ca2d84299a00842c3d7a7d0c67b55b0976b135e7f322"}
Jul 02 13:56:03.786053 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:56:03.786026    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj"
Jul 02 13:56:03.834689 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:56:03.834655    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2b91412d-7966-4d88-a431-ad3ac9b3041b-tokenizer-uds\") pod \"2b91412d-7966-4d88-a431-ad3ac9b3041b\" (UID: \"2b91412d-7966-4d88-a431-ad3ac9b3041b\") "
Jul 02 13:56:03.834862 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:56:03.834702    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b91412d-7966-4d88-a431-ad3ac9b3041b-tls-certs\") pod \"2b91412d-7966-4d88-a431-ad3ac9b3041b\" (UID: \"2b91412d-7966-4d88-a431-ad3ac9b3041b\") "
Jul 02 13:56:03.834862 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:56:03.834732    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdg8g\" (UniqueName: \"kubernetes.io/projected/2b91412d-7966-4d88-a431-ad3ac9b3041b-kube-api-access-sdg8g\") pod \"2b91412d-7966-4d88-a431-ad3ac9b3041b\" (UID: \"2b91412d-7966-4d88-a431-ad3ac9b3041b\") "
Jul 02 13:56:03.834862 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:56:03.834805    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b91412d-7966-4d88-a431-ad3ac9b3041b-kserve-provision-location\") pod \"2b91412d-7966-4d88-a431-ad3ac9b3041b\" (UID: \"2b91412d-7966-4d88-a431-ad3ac9b3041b\") "
Jul 02 13:56:03.834993 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:56:03.834949    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b91412d-7966-4d88-a431-ad3ac9b3041b-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "2b91412d-7966-4d88-a431-ad3ac9b3041b" (UID: "2b91412d-7966-4d88-a431-ad3ac9b3041b"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:56:03.835591 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:56:03.835561    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b91412d-7966-4d88-a431-ad3ac9b3041b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2b91412d-7966-4d88-a431-ad3ac9b3041b" (UID: "2b91412d-7966-4d88-a431-ad3ac9b3041b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:56:03.836944 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:56:03.836919    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b91412d-7966-4d88-a431-ad3ac9b3041b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2b91412d-7966-4d88-a431-ad3ac9b3041b" (UID: "2b91412d-7966-4d88-a431-ad3ac9b3041b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:56:03.837046 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:56:03.836978    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b91412d-7966-4d88-a431-ad3ac9b3041b-kube-api-access-sdg8g" (OuterVolumeSpecName: "kube-api-access-sdg8g") pod "2b91412d-7966-4d88-a431-ad3ac9b3041b" (UID: "2b91412d-7966-4d88-a431-ad3ac9b3041b"). InnerVolumeSpecName "kube-api-access-sdg8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:56:03.936471 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:56:03.936429    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdg8g\" (UniqueName: \"kubernetes.io/projected/2b91412d-7966-4d88-a431-ad3ac9b3041b-kube-api-access-sdg8g\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:56:03.936471 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:56:03.936462    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b91412d-7966-4d88-a431-ad3ac9b3041b-kserve-provision-location\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:56:03.936471 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:56:03.936473    2583 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2b91412d-7966-4d88-a431-ad3ac9b3041b-tokenizer-uds\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:56:03.936471 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:56:03.936484    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b91412d-7966-4d88-a431-ad3ac9b3041b-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 13:56:04.669040 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:56:04.669010    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj"
Jul 02 13:56:04.669040 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:56:04.669018    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj" event={"ID":"2b91412d-7966-4d88-a431-ad3ac9b3041b","Type":"ContainerDied","Data":"9a2efa1afbb31d6cf1e59f35cd34e3fda0413e31c90082ed9825d8c90bd161d4"}
Jul 02 13:56:04.669641 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:56:04.669081    2583 scope.go:117] "RemoveContainer" containerID="9c7f83b0bb3654c567f5ca2d84299a00842c3d7a7d0c67b55b0976b135e7f322"
Jul 02 13:56:04.678470 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:56:04.678444    2583 scope.go:117] "RemoveContainer" containerID="2154243c7a857bb93a06f866fa8bd36910b24cebc7410f2a4ba88daeafd2016f"
Jul 02 13:56:04.689930 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:56:04.689898    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj"]
Jul 02 13:56:04.693777 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:56:04.693747    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-58d77cc9cdfppj"]
Jul 02 13:56:05.234693 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:56:05.234648    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b91412d-7966-4d88-a431-ad3ac9b3041b" path="/var/lib/kubelet/pods/2b91412d-7966-4d88-a431-ad3ac9b3041b/volumes"
Jul 02 13:56:56.375768 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:56:56.375728    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:57:46.835746 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:57:46.835703    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:58:29.137151 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:58:29.137066    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:58:29.233440 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:58:29.233379    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:58:29.353781 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:58:29.353749    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:58:29.426481 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:58:29.426376    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t"]
Jul 02 13:58:29.426773 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:58:29.426760    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b91412d-7966-4d88-a431-ad3ac9b3041b" containerName="storage-initializer"
Jul 02 13:58:29.426817 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:58:29.426775    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b91412d-7966-4d88-a431-ad3ac9b3041b" containerName="storage-initializer"
Jul 02 13:58:29.426817 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:58:29.426787    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b91412d-7966-4d88-a431-ad3ac9b3041b" containerName="main"
Jul 02 13:58:29.426817 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:58:29.426792    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b91412d-7966-4d88-a431-ad3ac9b3041b" containerName="main"
Jul 02 13:58:29.426913 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:58:29.426851    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b91412d-7966-4d88-a431-ad3ac9b3041b" containerName="main"
Jul 02 13:58:29.429749 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:58:29.429726    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t"
Jul 02 13:58:29.433872 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:58:29.433846    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-dqvnm\""
Jul 02 13:58:29.433999 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:58:29.433872    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7s8sh\""
Jul 02 13:58:29.433999 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:58:29.433927    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 02 13:58:29.443901 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:58:29.443876    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t"]
Jul 02 13:58:29.465818 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:58:29.465781    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzr2d\" (UniqueName: \"kubernetes.io/projected/efdc86a8-bc40-445c-bd40-d607151733b7-kube-api-access-vzr2d\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t\" (UID: \"efdc86a8-bc40-445c-bd40-d607151733b7\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t"
Jul 02 13:58:29.466011 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:58:29.465838    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efdc86a8-bc40-445c-bd40-d607151733b7-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t\" (UID: \"efdc86a8-bc40-445c-bd40-d607151733b7\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t"
Jul 02 13:58:29.466011 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:58:29.465976    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/efdc86a8-bc40-445c-bd40-d607151733b7-tokenizer-uds\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t\" (UID: \"efdc86a8-bc40-445c-bd40-d607151733b7\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t"
Jul 02 13:58:29.467868 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:58:29.467839    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:58:29.567075 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:58:29.567035    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzr2d\" (UniqueName: \"kubernetes.io/projected/efdc86a8-bc40-445c-bd40-d607151733b7-kube-api-access-vzr2d\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t\" (UID: \"efdc86a8-bc40-445c-bd40-d607151733b7\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t"
Jul 02 13:58:29.567266 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:58:29.567090    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efdc86a8-bc40-445c-bd40-d607151733b7-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t\" (UID: \"efdc86a8-bc40-445c-bd40-d607151733b7\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t"
Jul 02 13:58:29.567266 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:58:29.567140    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/efdc86a8-bc40-445c-bd40-d607151733b7-tokenizer-uds\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t\" (UID: \"efdc86a8-bc40-445c-bd40-d607151733b7\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t"
Jul 02 13:58:29.567549 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:58:29.567530    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/efdc86a8-bc40-445c-bd40-d607151733b7-tokenizer-uds\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t\" (UID: \"efdc86a8-bc40-445c-bd40-d607151733b7\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t"
Jul 02 13:58:29.569751 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:58:29.569726    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efdc86a8-bc40-445c-bd40-d607151733b7-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t\" (UID: \"efdc86a8-bc40-445c-bd40-d607151733b7\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t"
Jul 02 13:58:29.575945 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:58:29.575917    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzr2d\" (UniqueName: \"kubernetes.io/projected/efdc86a8-bc40-445c-bd40-d607151733b7-kube-api-access-vzr2d\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t\" (UID: \"efdc86a8-bc40-445c-bd40-d607151733b7\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t"
Jul 02 13:58:29.739265 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:58:29.739174    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t"
Jul 02 13:58:29.876587 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:58:29.876559    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t"]
Jul 02 13:58:29.879145 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:58:29.879113    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefdc86a8_bc40_445c_bd40_d607151733b7.slice/crio-b5dbf7ee910125d56bfa9901a0552138e013d32ed709732d44b5774753b3c5db WatchSource:0}: Error finding container b5dbf7ee910125d56bfa9901a0552138e013d32ed709732d44b5774753b3c5db: Status 404 returned error can't find the container with id b5dbf7ee910125d56bfa9901a0552138e013d32ed709732d44b5774753b3c5db
Jul 02 13:58:30.389434 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:58:30.389397    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t" event={"ID":"efdc86a8-bc40-445c-bd40-d607151733b7","Type":"ContainerStarted","Data":"27576be0f69f0e4d430b8e489f3bea815744b71abd097846ce1bf43634d7e452"}
Jul 02 13:58:30.389793 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:58:30.389449    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t" event={"ID":"efdc86a8-bc40-445c-bd40-d607151733b7","Type":"ContainerStarted","Data":"b5dbf7ee910125d56bfa9901a0552138e013d32ed709732d44b5774753b3c5db"}
Jul 02 13:58:30.389793 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:58:30.389472    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t"
Jul 02 13:58:30.407945 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:58:30.407892    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t" podStartSLOduration=1.407875493 podStartE2EDuration="1.407875493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:58:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:58:30.406026333 +0000 UTC m=+2621.633937506" watchObservedRunningTime="2026-07-02 13:58:30.407875493 +0000 UTC m=+2621.635786667"
Jul 02 13:58:55.509606 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:58:55.509565    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:59:01.396343 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:01.396310    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t"
Jul 02 13:59:19.410134 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:19.410080    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:59:33.609616 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:33.609582    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:59:33.664236 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:33.664200    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:59:41.460049 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:41.459961    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:59:41.561509 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:41.561463    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:59:49.452483 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:49.452450    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 13:59:49.459990 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:49.459967    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 13:59:57.046489 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.046449    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:59:57.093784 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.093751    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 02 13:59:57.097491 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.097454    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 02 13:59:57.100133 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.100102    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 13:59:57.100133 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.100132    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-dqt4b\""
Jul 02 13:59:57.100342 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.100213    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 02 13:59:57.111108 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.111079    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 02 13:59:57.114913 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.114874    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:59:57.166739 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.166703    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/541a01ad-2163-47c8-9962-4c77d906de02-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"541a01ad-2163-47c8-9962-4c77d906de02\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 02 13:59:57.166917 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.166748    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/541a01ad-2163-47c8-9962-4c77d906de02-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"541a01ad-2163-47c8-9962-4c77d906de02\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 02 13:59:57.166917 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.166821    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/541a01ad-2163-47c8-9962-4c77d906de02-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"541a01ad-2163-47c8-9962-4c77d906de02\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 02 13:59:57.166917 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.166879    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/541a01ad-2163-47c8-9962-4c77d906de02-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"541a01ad-2163-47c8-9962-4c77d906de02\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 02 13:59:57.166917 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.166896    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/541a01ad-2163-47c8-9962-4c77d906de02-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"541a01ad-2163-47c8-9962-4c77d906de02\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 02 13:59:57.167089 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.166953    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/541a01ad-2163-47c8-9962-4c77d906de02-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"541a01ad-2163-47c8-9962-4c77d906de02\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 02 13:59:57.167089 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.166977    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2grjb\" (UniqueName: \"kubernetes.io/projected/541a01ad-2163-47c8-9962-4c77d906de02-kube-api-access-2grjb\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"541a01ad-2163-47c8-9962-4c77d906de02\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 02 13:59:57.268380 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.268346    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/541a01ad-2163-47c8-9962-4c77d906de02-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"541a01ad-2163-47c8-9962-4c77d906de02\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 02 13:59:57.268570 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.268413    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/541a01ad-2163-47c8-9962-4c77d906de02-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"541a01ad-2163-47c8-9962-4c77d906de02\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 02 13:59:57.268570 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.268478    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/541a01ad-2163-47c8-9962-4c77d906de02-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"541a01ad-2163-47c8-9962-4c77d906de02\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 02 13:59:57.268660 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.268586    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2grjb\" (UniqueName: \"kubernetes.io/projected/541a01ad-2163-47c8-9962-4c77d906de02-kube-api-access-2grjb\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"541a01ad-2163-47c8-9962-4c77d906de02\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 02 13:59:57.268660 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.268634    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/541a01ad-2163-47c8-9962-4c77d906de02-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"541a01ad-2163-47c8-9962-4c77d906de02\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 02 13:59:57.268741 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.268675    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/541a01ad-2163-47c8-9962-4c77d906de02-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"541a01ad-2163-47c8-9962-4c77d906de02\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 02 13:59:57.268794 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.268737    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/541a01ad-2163-47c8-9962-4c77d906de02-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"541a01ad-2163-47c8-9962-4c77d906de02\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 02 13:59:57.268918 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.268885    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/541a01ad-2163-47c8-9962-4c77d906de02-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"541a01ad-2163-47c8-9962-4c77d906de02\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 02 13:59:57.268918 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.268904    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/541a01ad-2163-47c8-9962-4c77d906de02-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"541a01ad-2163-47c8-9962-4c77d906de02\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 02 13:59:57.269078 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.269019    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/541a01ad-2163-47c8-9962-4c77d906de02-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"541a01ad-2163-47c8-9962-4c77d906de02\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 02 13:59:57.269122 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.269082    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/541a01ad-2163-47c8-9962-4c77d906de02-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"541a01ad-2163-47c8-9962-4c77d906de02\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 02 13:59:57.270842 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.270821    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/541a01ad-2163-47c8-9962-4c77d906de02-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"541a01ad-2163-47c8-9962-4c77d906de02\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 02 13:59:57.270842 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.270829    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/541a01ad-2163-47c8-9962-4c77d906de02-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"541a01ad-2163-47c8-9962-4c77d906de02\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 02 13:59:57.275955 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.275931    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2grjb\" (UniqueName: \"kubernetes.io/projected/541a01ad-2163-47c8-9962-4c77d906de02-kube-api-access-2grjb\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"541a01ad-2163-47c8-9962-4c77d906de02\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 02 13:59:57.314233 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.314135    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48"]
Jul 02 13:59:57.318157 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.318131    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48"
Jul 02 13:59:57.320806 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.320780    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-8lgct\""
Jul 02 13:59:57.327297 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.327273    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48"]
Jul 02 13:59:57.332562 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.332530    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 13:59:57.369506 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.369450    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c3df67b-64f5-4a86-af39-c1f670ccc437-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48\" (UID: \"7c3df67b-64f5-4a86-af39-c1f670ccc437\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48"
Jul 02 13:59:57.369690 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.369621    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djf6b\" (UniqueName: \"kubernetes.io/projected/7c3df67b-64f5-4a86-af39-c1f670ccc437-kube-api-access-djf6b\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48\" (UID: \"7c3df67b-64f5-4a86-af39-c1f670ccc437\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48"
Jul 02 13:59:57.369792 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.369765    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c3df67b-64f5-4a86-af39-c1f670ccc437-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48\" (UID: \"7c3df67b-64f5-4a86-af39-c1f670ccc437\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48"
Jul 02 13:59:57.369854 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.369809    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7c3df67b-64f5-4a86-af39-c1f670ccc437-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48\" (UID: \"7c3df67b-64f5-4a86-af39-c1f670ccc437\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48"
Jul 02 13:59:57.408142 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.408104    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 02 13:59:57.470765 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.470369    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djf6b\" (UniqueName: \"kubernetes.io/projected/7c3df67b-64f5-4a86-af39-c1f670ccc437-kube-api-access-djf6b\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48\" (UID: \"7c3df67b-64f5-4a86-af39-c1f670ccc437\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48"
Jul 02 13:59:57.470765 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.470502    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c3df67b-64f5-4a86-af39-c1f670ccc437-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48\" (UID: \"7c3df67b-64f5-4a86-af39-c1f670ccc437\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48"
Jul 02 13:59:57.470765 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.470531    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7c3df67b-64f5-4a86-af39-c1f670ccc437-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48\" (UID: \"7c3df67b-64f5-4a86-af39-c1f670ccc437\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48"
Jul 02 13:59:57.470765 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.470576    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c3df67b-64f5-4a86-af39-c1f670ccc437-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48\" (UID: \"7c3df67b-64f5-4a86-af39-c1f670ccc437\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48"
Jul 02 13:59:57.471025 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.471001    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c3df67b-64f5-4a86-af39-c1f670ccc437-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48\" (UID: \"7c3df67b-64f5-4a86-af39-c1f670ccc437\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48"
Jul 02 13:59:57.471501 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.471374    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7c3df67b-64f5-4a86-af39-c1f670ccc437-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48\" (UID: \"7c3df67b-64f5-4a86-af39-c1f670ccc437\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48"
Jul 02 13:59:57.474195 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.474141    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c3df67b-64f5-4a86-af39-c1f670ccc437-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48\" (UID: \"7c3df67b-64f5-4a86-af39-c1f670ccc437\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48"
Jul 02 13:59:57.480939 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.480909    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djf6b\" (UniqueName: \"kubernetes.io/projected/7c3df67b-64f5-4a86-af39-c1f670ccc437-kube-api-access-djf6b\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48\" (UID: \"7c3df67b-64f5-4a86-af39-c1f670ccc437\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48"
Jul 02 13:59:57.553550 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.553519    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 02 13:59:57.555684 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:59:57.555658    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod541a01ad_2163_47c8_9962_4c77d906de02.slice/crio-011502b23bd843a2d3b65bac8715cdb6bce072d80f44e634a19514c0666b7ef4 WatchSource:0}: Error finding container 011502b23bd843a2d3b65bac8715cdb6bce072d80f44e634a19514c0666b7ef4: Status 404 returned error can't find the container with id 011502b23bd843a2d3b65bac8715cdb6bce072d80f44e634a19514c0666b7ef4
Jul 02 13:59:57.628795 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.628759    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48"
Jul 02 13:59:57.776031 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.776004    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48"]
Jul 02 13:59:57.778502 ip-10-0-135-143 kubenswrapper[2583]: W0702 13:59:57.778462    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3df67b_64f5_4a86_af39_c1f670ccc437.slice/crio-e1c178ffe2947abf041893423e8c0b5c84a89187c1aacccb057a6647c9ca9247 WatchSource:0}: Error finding container e1c178ffe2947abf041893423e8c0b5c84a89187c1aacccb057a6647c9ca9247: Status 404 returned error can't find the container with id e1c178ffe2947abf041893423e8c0b5c84a89187c1aacccb057a6647c9ca9247
Jul 02 13:59:57.826329 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.826294    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"541a01ad-2163-47c8-9962-4c77d906de02","Type":"ContainerStarted","Data":"0cd1df5272fbb9a96504b31c81780130fd7f85a12e4f2b9cd2008f5dd78b50b9"}
Jul 02 13:59:57.826533 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.826357    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"541a01ad-2163-47c8-9962-4c77d906de02","Type":"ContainerStarted","Data":"011502b23bd843a2d3b65bac8715cdb6bce072d80f44e634a19514c0666b7ef4"}
Jul 02 13:59:57.827542 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:57.827513    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48" event={"ID":"7c3df67b-64f5-4a86-af39-c1f670ccc437","Type":"ContainerStarted","Data":"e1c178ffe2947abf041893423e8c0b5c84a89187c1aacccb057a6647c9ca9247"}
Jul 02 13:59:58.835467 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:58.835355    2583 generic.go:356] "Generic (PLEG): container finished" podID="7c3df67b-64f5-4a86-af39-c1f670ccc437" containerID="a92b1f172b4910bc215040f2062be99e6a3e3575127e2a444bcea9747694b5b6" exitCode=0
Jul 02 13:59:58.835467 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:58.835450    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48" event={"ID":"7c3df67b-64f5-4a86-af39-c1f670ccc437","Type":"ContainerDied","Data":"a92b1f172b4910bc215040f2062be99e6a3e3575127e2a444bcea9747694b5b6"}
Jul 02 13:59:59.842334 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:59.842288    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48" event={"ID":"7c3df67b-64f5-4a86-af39-c1f670ccc437","Type":"ContainerStarted","Data":"864d4391b3aa35b644fd06eda60fc2901fad852d9c7359aa620c4c282b649016"}
Jul 02 13:59:59.842736 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:59.842366    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48"
Jul 02 13:59:59.864078 ip-10-0-135-143 kubenswrapper[2583]: I0702 13:59:59.864019    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48" podStartSLOduration=2.162665436 podStartE2EDuration="2.864000034s" totalImagesPullingTime="0s" totalInitContainerRuntime="701.334598ms" isStatefulPod=false podCreationTimestamp="2026-07-02 13:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:59:59.861890498 +0000 UTC m=+2711.089801674" watchObservedRunningTime="2026-07-02 13:59:59.864000034 +0000 UTC m=+2711.091911226"
Jul 02 14:00:02.860457 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:00:02.860412    2583 generic.go:356] "Generic (PLEG): container finished" podID="541a01ad-2163-47c8-9962-4c77d906de02" containerID="0cd1df5272fbb9a96504b31c81780130fd7f85a12e4f2b9cd2008f5dd78b50b9" exitCode=0
Jul 02 14:00:02.860838 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:00:02.860478    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"541a01ad-2163-47c8-9962-4c77d906de02","Type":"ContainerDied","Data":"0cd1df5272fbb9a96504b31c81780130fd7f85a12e4f2b9cd2008f5dd78b50b9"}
Jul 02 14:00:02.861846 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:00:02.861829    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 14:00:30.850726 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:00:30.850693    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48"
Jul 02 14:00:33.060520 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:00:33.060423    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"541a01ad-2163-47c8-9962-4c77d906de02","Type":"ContainerStarted","Data":"f1b41d5994c8256fe3193ff360fa643e32aef1c8d0c630c861c7b0ddcb93557e"}
Jul 02 14:00:33.082356 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:00:33.082301    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=1.8314352249999999 podStartE2EDuration="36.08228471s" totalImagesPullingTime="29.774390191s" totalInitContainerRuntime="4.476459294s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:59:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:00:33.080137376 +0000 UTC m=+2744.308048563" watchObservedRunningTime="2026-07-02 14:00:33.08228471 +0000 UTC m=+2744.310195884"
Jul 02 14:00:33.504282 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:00:33.504247    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:03:28.499016 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:03:28.498970    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:04:34.206532 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:04:34.206419    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:04:49.485319 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:04:49.485288    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 14:04:49.493621 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:04:49.493599    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 14:07:38.916787 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:07:38.916740    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:09:37.368597 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:09:37.368556    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:09:49.525950 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:09:49.525919    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 14:09:49.535113 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:09:49.535087    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 14:10:57.952012 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:10:57.951976    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:11:25.558204 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:11:25.558165    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:11:25.741003 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:11:25.740966    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:11:31.865825 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:11:31.865784    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:11:31.976479 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:11:31.976435    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:11:36.993235 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:11:36.993189    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:11:37.092891 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:11:37.092850    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:12:00.178936 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:00.178846    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:12:28.671597 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:28.671564    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:12:28.753681 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:28.753642    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:12:31.274864 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:31.274821    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:12:31.498097 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:31.498068    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:12:33.294339 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.294305    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["cert-manager/cert-manager-59c77684cc-smcs9"]
Jul 02 14:12:33.294798 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.294527    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="cert-manager/cert-manager-59c77684cc-smcs9" podUID="f7e09362-f89d-493f-bcf7-b7b70e6fe602" containerName="cert-manager-controller" containerID="cri-o://9935a5f8591db9f23f1bd20ecc16e2a40e47954531576b634b5495186c843b77" gracePeriod=30
Jul 02 14:12:33.312848 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.312811    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["cert-manager/cert-manager-cainjector-5d99449ffd-b8llt"]
Jul 02 14:12:33.313069 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.313043    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="cert-manager/cert-manager-cainjector-5d99449ffd-b8llt" podUID="5719b470-94d0-4677-bb02-40d737e1701c" containerName="cert-manager-cainjector" containerID="cri-o://9133aad2e49007b8ec97e40e7fc314b46a1c484881b71d4cfbe2696e0e0251c5" gracePeriod=30
Jul 02 14:12:33.581327 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.581303    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-59c77684cc-smcs9"
Jul 02 14:12:33.584367 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.584347    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5d99449ffd-b8llt"
Jul 02 14:12:33.644548 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.644498    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pzk4\" (UniqueName: \"kubernetes.io/projected/f7e09362-f89d-493f-bcf7-b7b70e6fe602-kube-api-access-9pzk4\") pod \"f7e09362-f89d-493f-bcf7-b7b70e6fe602\" (UID: \"f7e09362-f89d-493f-bcf7-b7b70e6fe602\") "
Jul 02 14:12:33.644548 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.644554    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7e09362-f89d-493f-bcf7-b7b70e6fe602-bound-sa-token\") pod \"f7e09362-f89d-493f-bcf7-b7b70e6fe602\" (UID: \"f7e09362-f89d-493f-bcf7-b7b70e6fe602\") "
Jul 02 14:12:33.644782 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.644579    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5719b470-94d0-4677-bb02-40d737e1701c-bound-sa-token\") pod \"5719b470-94d0-4677-bb02-40d737e1701c\" (UID: \"5719b470-94d0-4677-bb02-40d737e1701c\") "
Jul 02 14:12:33.644782 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.644597    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9r5fp\" (UniqueName: \"kubernetes.io/projected/5719b470-94d0-4677-bb02-40d737e1701c-kube-api-access-9r5fp\") pod \"5719b470-94d0-4677-bb02-40d737e1701c\" (UID: \"5719b470-94d0-4677-bb02-40d737e1701c\") "
Jul 02 14:12:33.646954 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.646918    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7e09362-f89d-493f-bcf7-b7b70e6fe602-kube-api-access-9pzk4" (OuterVolumeSpecName: "kube-api-access-9pzk4") pod "f7e09362-f89d-493f-bcf7-b7b70e6fe602" (UID: "f7e09362-f89d-493f-bcf7-b7b70e6fe602"). InnerVolumeSpecName "kube-api-access-9pzk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:12:33.646954 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.646935    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5719b470-94d0-4677-bb02-40d737e1701c-kube-api-access-9r5fp" (OuterVolumeSpecName: "kube-api-access-9r5fp") pod "5719b470-94d0-4677-bb02-40d737e1701c" (UID: "5719b470-94d0-4677-bb02-40d737e1701c"). InnerVolumeSpecName "kube-api-access-9r5fp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:12:33.646954 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.646918    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7e09362-f89d-493f-bcf7-b7b70e6fe602-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f7e09362-f89d-493f-bcf7-b7b70e6fe602" (UID: "f7e09362-f89d-493f-bcf7-b7b70e6fe602"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:12:33.647140 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.646933    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5719b470-94d0-4677-bb02-40d737e1701c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5719b470-94d0-4677-bb02-40d737e1701c" (UID: "5719b470-94d0-4677-bb02-40d737e1701c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:12:33.745705 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.745654    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pzk4\" (UniqueName: \"kubernetes.io/projected/f7e09362-f89d-493f-bcf7-b7b70e6fe602-kube-api-access-9pzk4\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:12:33.745705 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.745702    2583 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7e09362-f89d-493f-bcf7-b7b70e6fe602-bound-sa-token\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:12:33.745705 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.745711    2583 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5719b470-94d0-4677-bb02-40d737e1701c-bound-sa-token\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:12:33.745705 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.745719    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r5fp\" (UniqueName: \"kubernetes.io/projected/5719b470-94d0-4677-bb02-40d737e1701c-kube-api-access-9r5fp\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:12:33.790658 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.790623    2583 generic.go:356] "Generic (PLEG): container finished" podID="5719b470-94d0-4677-bb02-40d737e1701c" containerID="9133aad2e49007b8ec97e40e7fc314b46a1c484881b71d4cfbe2696e0e0251c5" exitCode=0
Jul 02 14:12:33.790857 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.790672    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5d99449ffd-b8llt"
Jul 02 14:12:33.790857 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.790715    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5d99449ffd-b8llt" event={"ID":"5719b470-94d0-4677-bb02-40d737e1701c","Type":"ContainerDied","Data":"9133aad2e49007b8ec97e40e7fc314b46a1c484881b71d4cfbe2696e0e0251c5"}
Jul 02 14:12:33.790857 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.790767    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5d99449ffd-b8llt" event={"ID":"5719b470-94d0-4677-bb02-40d737e1701c","Type":"ContainerDied","Data":"05457f56fee45eaeea7e78fe96c4c183aab8fb32d6d62db9c37691e811effa3b"}
Jul 02 14:12:33.790857 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.790791    2583 scope.go:117] "RemoveContainer" containerID="9133aad2e49007b8ec97e40e7fc314b46a1c484881b71d4cfbe2696e0e0251c5"
Jul 02 14:12:33.792065 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.792042    2583 generic.go:356] "Generic (PLEG): container finished" podID="f7e09362-f89d-493f-bcf7-b7b70e6fe602" containerID="9935a5f8591db9f23f1bd20ecc16e2a40e47954531576b634b5495186c843b77" exitCode=0
Jul 02 14:12:33.792142 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.792083    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-59c77684cc-smcs9"
Jul 02 14:12:33.792142 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.792122    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-59c77684cc-smcs9" event={"ID":"f7e09362-f89d-493f-bcf7-b7b70e6fe602","Type":"ContainerDied","Data":"9935a5f8591db9f23f1bd20ecc16e2a40e47954531576b634b5495186c843b77"}
Jul 02 14:12:33.792318 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.792158    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-59c77684cc-smcs9" event={"ID":"f7e09362-f89d-493f-bcf7-b7b70e6fe602","Type":"ContainerDied","Data":"3a699c2ccab83002b835194211f347540b365d08b81a9a4e7291af51e95f49ad"}
Jul 02 14:12:33.799937 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.799918    2583 scope.go:117] "RemoveContainer" containerID="9133aad2e49007b8ec97e40e7fc314b46a1c484881b71d4cfbe2696e0e0251c5"
Jul 02 14:12:33.800332 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:12:33.800199    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9133aad2e49007b8ec97e40e7fc314b46a1c484881b71d4cfbe2696e0e0251c5\": container with ID starting with 9133aad2e49007b8ec97e40e7fc314b46a1c484881b71d4cfbe2696e0e0251c5 not found: ID does not exist" containerID="9133aad2e49007b8ec97e40e7fc314b46a1c484881b71d4cfbe2696e0e0251c5"
Jul 02 14:12:33.800332 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.800230    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9133aad2e49007b8ec97e40e7fc314b46a1c484881b71d4cfbe2696e0e0251c5"} err="failed to get container status \"9133aad2e49007b8ec97e40e7fc314b46a1c484881b71d4cfbe2696e0e0251c5\": rpc error: code = NotFound desc = could not find container \"9133aad2e49007b8ec97e40e7fc314b46a1c484881b71d4cfbe2696e0e0251c5\": container with ID starting with 9133aad2e49007b8ec97e40e7fc314b46a1c484881b71d4cfbe2696e0e0251c5 not found: ID does not exist"
Jul 02 14:12:33.800332 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.800252    2583 scope.go:117] "RemoveContainer" containerID="9935a5f8591db9f23f1bd20ecc16e2a40e47954531576b634b5495186c843b77"
Jul 02 14:12:33.808688 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.808668    2583 scope.go:117] "RemoveContainer" containerID="9935a5f8591db9f23f1bd20ecc16e2a40e47954531576b634b5495186c843b77"
Jul 02 14:12:33.808957 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:12:33.808938    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9935a5f8591db9f23f1bd20ecc16e2a40e47954531576b634b5495186c843b77\": container with ID starting with 9935a5f8591db9f23f1bd20ecc16e2a40e47954531576b634b5495186c843b77 not found: ID does not exist" containerID="9935a5f8591db9f23f1bd20ecc16e2a40e47954531576b634b5495186c843b77"
Jul 02 14:12:33.809020 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.808964    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9935a5f8591db9f23f1bd20ecc16e2a40e47954531576b634b5495186c843b77"} err="failed to get container status \"9935a5f8591db9f23f1bd20ecc16e2a40e47954531576b634b5495186c843b77\": rpc error: code = NotFound desc = could not find container \"9935a5f8591db9f23f1bd20ecc16e2a40e47954531576b634b5495186c843b77\": container with ID starting with 9935a5f8591db9f23f1bd20ecc16e2a40e47954531576b634b5495186c843b77 not found: ID does not exist"
Jul 02 14:12:33.815153 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.815118    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["cert-manager/cert-manager-59c77684cc-smcs9"]
Jul 02 14:12:33.819486 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.819458    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["cert-manager/cert-manager-59c77684cc-smcs9"]
Jul 02 14:12:33.829955 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.829925    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["cert-manager/cert-manager-cainjector-5d99449ffd-b8llt"]
Jul 02 14:12:33.831676 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:33.831656    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["cert-manager/cert-manager-cainjector-5d99449ffd-b8llt"]
Jul 02 14:12:35.234345 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:35.234311    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5719b470-94d0-4677-bb02-40d737e1701c" path="/var/lib/kubelet/pods/5719b470-94d0-4677-bb02-40d737e1701c/volumes"
Jul 02 14:12:35.234712 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:35.234670    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7e09362-f89d-493f-bcf7-b7b70e6fe602" path="/var/lib/kubelet/pods/f7e09362-f89d-493f-bcf7-b7b70e6fe602/volumes"
Jul 02 14:12:38.270639 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:38.270600    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:12:39.276503 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:39.276466    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["cert-manager/cert-manager-webhook-7dc76cbb94-6x6wr"]
Jul 02 14:12:39.276921 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:39.276675    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="cert-manager/cert-manager-webhook-7dc76cbb94-6x6wr" podUID="b50537c9-7f32-4af5-9b58-062e9b0ba7b7" containerName="cert-manager-webhook" containerID="cri-o://fc8c58426f9dab48a49a3919c0263fcba2255b67d34b498b1526df207f53468f" gracePeriod=30
Jul 02 14:12:39.536298 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:39.536235    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-7dc76cbb94-6x6wr"
Jul 02 14:12:39.598778 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:39.598743    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftrfh\" (UniqueName: \"kubernetes.io/projected/b50537c9-7f32-4af5-9b58-062e9b0ba7b7-kube-api-access-ftrfh\") pod \"b50537c9-7f32-4af5-9b58-062e9b0ba7b7\" (UID: \"b50537c9-7f32-4af5-9b58-062e9b0ba7b7\") "
Jul 02 14:12:39.598948 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:39.598917    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b50537c9-7f32-4af5-9b58-062e9b0ba7b7-bound-sa-token\") pod \"b50537c9-7f32-4af5-9b58-062e9b0ba7b7\" (UID: \"b50537c9-7f32-4af5-9b58-062e9b0ba7b7\") "
Jul 02 14:12:39.600871 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:39.600842    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b50537c9-7f32-4af5-9b58-062e9b0ba7b7-kube-api-access-ftrfh" (OuterVolumeSpecName: "kube-api-access-ftrfh") pod "b50537c9-7f32-4af5-9b58-062e9b0ba7b7" (UID: "b50537c9-7f32-4af5-9b58-062e9b0ba7b7"). InnerVolumeSpecName "kube-api-access-ftrfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:12:39.600996 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:39.600971    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b50537c9-7f32-4af5-9b58-062e9b0ba7b7-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b50537c9-7f32-4af5-9b58-062e9b0ba7b7" (UID: "b50537c9-7f32-4af5-9b58-062e9b0ba7b7"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:12:39.700327 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:39.700287    2583 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b50537c9-7f32-4af5-9b58-062e9b0ba7b7-bound-sa-token\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:12:39.700327 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:39.700318    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftrfh\" (UniqueName: \"kubernetes.io/projected/b50537c9-7f32-4af5-9b58-062e9b0ba7b7-kube-api-access-ftrfh\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:12:39.825095 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:39.825003    2583 generic.go:356] "Generic (PLEG): container finished" podID="b50537c9-7f32-4af5-9b58-062e9b0ba7b7" containerID="fc8c58426f9dab48a49a3919c0263fcba2255b67d34b498b1526df207f53468f" exitCode=0
Jul 02 14:12:39.825095 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:39.825043    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-7dc76cbb94-6x6wr"
Jul 02 14:12:39.825095 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:39.825077    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-7dc76cbb94-6x6wr" event={"ID":"b50537c9-7f32-4af5-9b58-062e9b0ba7b7","Type":"ContainerDied","Data":"fc8c58426f9dab48a49a3919c0263fcba2255b67d34b498b1526df207f53468f"}
Jul 02 14:12:39.825326 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:39.825107    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-7dc76cbb94-6x6wr" event={"ID":"b50537c9-7f32-4af5-9b58-062e9b0ba7b7","Type":"ContainerDied","Data":"5b729536d46e5410078dc115910cd89a00a29ce7206bd988e9a634c43056da69"}
Jul 02 14:12:39.825326 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:39.825124    2583 scope.go:117] "RemoveContainer" containerID="fc8c58426f9dab48a49a3919c0263fcba2255b67d34b498b1526df207f53468f"
Jul 02 14:12:39.834628 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:39.834610    2583 scope.go:117] "RemoveContainer" containerID="fc8c58426f9dab48a49a3919c0263fcba2255b67d34b498b1526df207f53468f"
Jul 02 14:12:39.834898 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:12:39.834878    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc8c58426f9dab48a49a3919c0263fcba2255b67d34b498b1526df207f53468f\": container with ID starting with fc8c58426f9dab48a49a3919c0263fcba2255b67d34b498b1526df207f53468f not found: ID does not exist" containerID="fc8c58426f9dab48a49a3919c0263fcba2255b67d34b498b1526df207f53468f"
Jul 02 14:12:39.834957 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:39.834907    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc8c58426f9dab48a49a3919c0263fcba2255b67d34b498b1526df207f53468f"} err="failed to get container status \"fc8c58426f9dab48a49a3919c0263fcba2255b67d34b498b1526df207f53468f\": rpc error: code = NotFound desc = could not find container \"fc8c58426f9dab48a49a3919c0263fcba2255b67d34b498b1526df207f53468f\": container with ID starting with fc8c58426f9dab48a49a3919c0263fcba2255b67d34b498b1526df207f53468f not found: ID does not exist"
Jul 02 14:12:39.846435 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:39.846407    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["cert-manager/cert-manager-webhook-7dc76cbb94-6x6wr"]
Jul 02 14:12:39.850378 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:39.850355    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["cert-manager/cert-manager-webhook-7dc76cbb94-6x6wr"]
Jul 02 14:12:41.233584 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:12:41.233550    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b50537c9-7f32-4af5-9b58-062e9b0ba7b7" path="/var/lib/kubelet/pods/b50537c9-7f32-4af5-9b58-062e9b0ba7b7/volumes"
Jul 02 14:13:21.488145 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:21.488064    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t_efdc86a8-bc40-445c-bd40-d607151733b7/main/0.log"
Jul 02 14:13:54.699537 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.699501    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"]
Jul 02 14:13:54.700043 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.699912    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b50537c9-7f32-4af5-9b58-062e9b0ba7b7" containerName="cert-manager-webhook"
Jul 02 14:13:54.700043 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.699925    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b50537c9-7f32-4af5-9b58-062e9b0ba7b7" containerName="cert-manager-webhook"
Jul 02 14:13:54.700043 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.699944    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7e09362-f89d-493f-bcf7-b7b70e6fe602" containerName="cert-manager-controller"
Jul 02 14:13:54.700043 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.699949    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7e09362-f89d-493f-bcf7-b7b70e6fe602" containerName="cert-manager-controller"
Jul 02 14:13:54.700043 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.699958    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5719b470-94d0-4677-bb02-40d737e1701c" containerName="cert-manager-cainjector"
Jul 02 14:13:54.700043 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.699964    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5719b470-94d0-4677-bb02-40d737e1701c" containerName="cert-manager-cainjector"
Jul 02 14:13:54.700043 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.700047    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5719b470-94d0-4677-bb02-40d737e1701c" containerName="cert-manager-cainjector"
Jul 02 14:13:54.700289 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.700055    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b50537c9-7f32-4af5-9b58-062e9b0ba7b7" containerName="cert-manager-webhook"
Jul 02 14:13:54.700289 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.700061    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7e09362-f89d-493f-bcf7-b7b70e6fe602" containerName="cert-manager-controller"
Jul 02 14:13:54.702193 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.702173    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"
Jul 02 14:13:54.704968 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.704947    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 02 14:13:54.716908 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.716878    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"]
Jul 02 14:13:54.739250 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.739217    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:13:54.809549 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.809501    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2856e2ae-2d5a-4936-a71a-1b9047b708a9-home\") pod \"stop-feature-test-kserve-757c67794-bpbsh\" (UID: \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"
Jul 02 14:13:54.809724 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.809556    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2856e2ae-2d5a-4936-a71a-1b9047b708a9-model-cache\") pod \"stop-feature-test-kserve-757c67794-bpbsh\" (UID: \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"
Jul 02 14:13:54.809724 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.809579    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgqc9\" (UniqueName: \"kubernetes.io/projected/2856e2ae-2d5a-4936-a71a-1b9047b708a9-kube-api-access-xgqc9\") pod \"stop-feature-test-kserve-757c67794-bpbsh\" (UID: \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"
Jul 02 14:13:54.809724 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.809660    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2856e2ae-2d5a-4936-a71a-1b9047b708a9-tmp-dir\") pod \"stop-feature-test-kserve-757c67794-bpbsh\" (UID: \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"
Jul 02 14:13:54.809724 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.809716    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2856e2ae-2d5a-4936-a71a-1b9047b708a9-dshm\") pod \"stop-feature-test-kserve-757c67794-bpbsh\" (UID: \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"
Jul 02 14:13:54.809866 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.809743    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2856e2ae-2d5a-4936-a71a-1b9047b708a9-kserve-provision-location\") pod \"stop-feature-test-kserve-757c67794-bpbsh\" (UID: \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"
Jul 02 14:13:54.809866 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.809763    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2856e2ae-2d5a-4936-a71a-1b9047b708a9-tls-certs\") pod \"stop-feature-test-kserve-757c67794-bpbsh\" (UID: \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"
Jul 02 14:13:54.911158 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.911114    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2856e2ae-2d5a-4936-a71a-1b9047b708a9-home\") pod \"stop-feature-test-kserve-757c67794-bpbsh\" (UID: \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"
Jul 02 14:13:54.911348 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.911170    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2856e2ae-2d5a-4936-a71a-1b9047b708a9-model-cache\") pod \"stop-feature-test-kserve-757c67794-bpbsh\" (UID: \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"
Jul 02 14:13:54.911348 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.911201    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgqc9\" (UniqueName: \"kubernetes.io/projected/2856e2ae-2d5a-4936-a71a-1b9047b708a9-kube-api-access-xgqc9\") pod \"stop-feature-test-kserve-757c67794-bpbsh\" (UID: \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"
Jul 02 14:13:54.911348 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.911248    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2856e2ae-2d5a-4936-a71a-1b9047b708a9-tmp-dir\") pod \"stop-feature-test-kserve-757c67794-bpbsh\" (UID: \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"
Jul 02 14:13:54.911348 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.911304    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2856e2ae-2d5a-4936-a71a-1b9047b708a9-dshm\") pod \"stop-feature-test-kserve-757c67794-bpbsh\" (UID: \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"
Jul 02 14:13:54.911348 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.911331    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2856e2ae-2d5a-4936-a71a-1b9047b708a9-kserve-provision-location\") pod \"stop-feature-test-kserve-757c67794-bpbsh\" (UID: \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"
Jul 02 14:13:54.911661 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.911355    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2856e2ae-2d5a-4936-a71a-1b9047b708a9-tls-certs\") pod \"stop-feature-test-kserve-757c67794-bpbsh\" (UID: \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"
Jul 02 14:13:54.911723 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.911654    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2856e2ae-2d5a-4936-a71a-1b9047b708a9-home\") pod \"stop-feature-test-kserve-757c67794-bpbsh\" (UID: \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"
Jul 02 14:13:54.911723 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.911706    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2856e2ae-2d5a-4936-a71a-1b9047b708a9-model-cache\") pod \"stop-feature-test-kserve-757c67794-bpbsh\" (UID: \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"
Jul 02 14:13:54.911826 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.911725    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2856e2ae-2d5a-4936-a71a-1b9047b708a9-tmp-dir\") pod \"stop-feature-test-kserve-757c67794-bpbsh\" (UID: \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"
Jul 02 14:13:54.911826 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.911751    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2856e2ae-2d5a-4936-a71a-1b9047b708a9-kserve-provision-location\") pod \"stop-feature-test-kserve-757c67794-bpbsh\" (UID: \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"
Jul 02 14:13:54.913873 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.913851    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2856e2ae-2d5a-4936-a71a-1b9047b708a9-dshm\") pod \"stop-feature-test-kserve-757c67794-bpbsh\" (UID: \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"
Jul 02 14:13:54.913987 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.913887    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2856e2ae-2d5a-4936-a71a-1b9047b708a9-tls-certs\") pod \"stop-feature-test-kserve-757c67794-bpbsh\" (UID: \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"
Jul 02 14:13:54.920437 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:54.920404    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgqc9\" (UniqueName: \"kubernetes.io/projected/2856e2ae-2d5a-4936-a71a-1b9047b708a9-kube-api-access-xgqc9\") pod \"stop-feature-test-kserve-757c67794-bpbsh\" (UID: \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"
Jul 02 14:13:55.012574 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:55.012486    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9"]
Jul 02 14:13:55.012725 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:55.012610    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"
Jul 02 14:13:55.016092 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:55.016069    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9"
Jul 02 14:13:55.018977 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:55.018957    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-b5px8\""
Jul 02 14:13:55.029167 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:55.027942    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9"]
Jul 02 14:13:55.052009 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:55.051973    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:13:55.114207 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:55.114133    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99157bb7-73e8-466f-ad6a-a799dffee0a2-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9\" (UID: \"99157bb7-73e8-466f-ad6a-a799dffee0a2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9"
Jul 02 14:13:55.114207 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:55.114193    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/99157bb7-73e8-466f-ad6a-a799dffee0a2-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9\" (UID: \"99157bb7-73e8-466f-ad6a-a799dffee0a2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9"
Jul 02 14:13:55.114472 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:55.114254    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2nqz\" (UniqueName: \"kubernetes.io/projected/99157bb7-73e8-466f-ad6a-a799dffee0a2-kube-api-access-n2nqz\") pod \"stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9\" (UID: \"99157bb7-73e8-466f-ad6a-a799dffee0a2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9"
Jul 02 14:13:55.114472 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:55.114292    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99157bb7-73e8-466f-ad6a-a799dffee0a2-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9\" (UID: \"99157bb7-73e8-466f-ad6a-a799dffee0a2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9"
Jul 02 14:13:55.171591 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:55.171547    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"]
Jul 02 14:13:55.175486 ip-10-0-135-143 kubenswrapper[2583]: W0702 14:13:55.175446    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2856e2ae_2d5a_4936_a71a_1b9047b708a9.slice/crio-80f8786651db9ecff5e08f173cc5f0b784ab71c5e4c2ab93392a745824cabeb6 WatchSource:0}: Error finding container 80f8786651db9ecff5e08f173cc5f0b784ab71c5e4c2ab93392a745824cabeb6: Status 404 returned error can't find the container with id 80f8786651db9ecff5e08f173cc5f0b784ab71c5e4c2ab93392a745824cabeb6
Jul 02 14:13:55.179701 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:55.179679    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 14:13:55.199200 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:55.199171    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh" event={"ID":"2856e2ae-2d5a-4936-a71a-1b9047b708a9","Type":"ContainerStarted","Data":"80f8786651db9ecff5e08f173cc5f0b784ab71c5e4c2ab93392a745824cabeb6"}
Jul 02 14:13:55.215700 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:55.215668    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2nqz\" (UniqueName: \"kubernetes.io/projected/99157bb7-73e8-466f-ad6a-a799dffee0a2-kube-api-access-n2nqz\") pod \"stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9\" (UID: \"99157bb7-73e8-466f-ad6a-a799dffee0a2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9"
Jul 02 14:13:55.215858 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:55.215710    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99157bb7-73e8-466f-ad6a-a799dffee0a2-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9\" (UID: \"99157bb7-73e8-466f-ad6a-a799dffee0a2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9"
Jul 02 14:13:55.215858 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:55.215827    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99157bb7-73e8-466f-ad6a-a799dffee0a2-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9\" (UID: \"99157bb7-73e8-466f-ad6a-a799dffee0a2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9"
Jul 02 14:13:55.215858 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:55.215852    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/99157bb7-73e8-466f-ad6a-a799dffee0a2-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9\" (UID: \"99157bb7-73e8-466f-ad6a-a799dffee0a2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9"
Jul 02 14:13:55.216160 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:55.216141    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99157bb7-73e8-466f-ad6a-a799dffee0a2-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9\" (UID: \"99157bb7-73e8-466f-ad6a-a799dffee0a2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9"
Jul 02 14:13:55.216257 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:55.216234    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/99157bb7-73e8-466f-ad6a-a799dffee0a2-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9\" (UID: \"99157bb7-73e8-466f-ad6a-a799dffee0a2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9"
Jul 02 14:13:55.218518 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:55.218494    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99157bb7-73e8-466f-ad6a-a799dffee0a2-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9\" (UID: \"99157bb7-73e8-466f-ad6a-a799dffee0a2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9"
Jul 02 14:13:55.225745 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:55.225716    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2nqz\" (UniqueName: \"kubernetes.io/projected/99157bb7-73e8-466f-ad6a-a799dffee0a2-kube-api-access-n2nqz\") pod \"stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9\" (UID: \"99157bb7-73e8-466f-ad6a-a799dffee0a2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9"
Jul 02 14:13:55.330563 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:55.330466    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9"
Jul 02 14:13:55.483500 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:55.483465    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9"]
Jul 02 14:13:55.484739 ip-10-0-135-143 kubenswrapper[2583]: W0702 14:13:55.484710    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99157bb7_73e8_466f_ad6a_a799dffee0a2.slice/crio-2534cad8c9111d87e8e4b039a91cda26af553797ae8bd9fabb29987cd3a824c7 WatchSource:0}: Error finding container 2534cad8c9111d87e8e4b039a91cda26af553797ae8bd9fabb29987cd3a824c7: Status 404 returned error can't find the container with id 2534cad8c9111d87e8e4b039a91cda26af553797ae8bd9fabb29987cd3a824c7
Jul 02 14:13:56.207375 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:56.207332    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh" event={"ID":"2856e2ae-2d5a-4936-a71a-1b9047b708a9","Type":"ContainerStarted","Data":"495fb531a9ac43e6c9b855ce5e7b307018cc7ecde4f5c4e52747af239daad572"}
Jul 02 14:13:56.210041 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:56.210005    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9" event={"ID":"99157bb7-73e8-466f-ad6a-a799dffee0a2","Type":"ContainerStarted","Data":"eb6b368ec0a9363daab46b75a4e39ee07bfdfcd5d3d60f1ac1439e24487cf002"}
Jul 02 14:13:56.210186 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:56.210067    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9" event={"ID":"99157bb7-73e8-466f-ad6a-a799dffee0a2","Type":"ContainerStarted","Data":"2534cad8c9111d87e8e4b039a91cda26af553797ae8bd9fabb29987cd3a824c7"}
Jul 02 14:13:57.217516 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:57.217472    2583 generic.go:356] "Generic (PLEG): container finished" podID="99157bb7-73e8-466f-ad6a-a799dffee0a2" containerID="eb6b368ec0a9363daab46b75a4e39ee07bfdfcd5d3d60f1ac1439e24487cf002" exitCode=0
Jul 02 14:13:57.217949 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:57.217564    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9" event={"ID":"99157bb7-73e8-466f-ad6a-a799dffee0a2","Type":"ContainerDied","Data":"eb6b368ec0a9363daab46b75a4e39ee07bfdfcd5d3d60f1ac1439e24487cf002"}
Jul 02 14:13:58.224854 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:58.224815    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9" event={"ID":"99157bb7-73e8-466f-ad6a-a799dffee0a2","Type":"ContainerStarted","Data":"f57d850ec97fabc6fbfa4acc76bd2684375f0e6f34ccd0f98c48716f70d97cf3"}
Jul 02 14:13:58.225238 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:58.224878    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9"
Jul 02 14:13:58.248366 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:13:58.248311    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9" podStartSLOduration=3.498478022 podStartE2EDuration="4.248294256s" totalImagesPullingTime="0s" totalInitContainerRuntime="749.816234ms" isStatefulPod=false podCreationTimestamp="2026-07-02 14:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:13:58.246123085 +0000 UTC m=+3549.474034262" watchObservedRunningTime="2026-07-02 14:13:58.248294256 +0000 UTC m=+3549.476205437"
Jul 02 14:14:00.236720 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:14:00.236687    2583 generic.go:356] "Generic (PLEG): container finished" podID="2856e2ae-2d5a-4936-a71a-1b9047b708a9" containerID="495fb531a9ac43e6c9b855ce5e7b307018cc7ecde4f5c4e52747af239daad572" exitCode=0
Jul 02 14:14:00.237106 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:14:00.236746    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh" event={"ID":"2856e2ae-2d5a-4936-a71a-1b9047b708a9","Type":"ContainerDied","Data":"495fb531a9ac43e6c9b855ce5e7b307018cc7ecde4f5c4e52747af239daad572"}
Jul 02 14:14:01.244938 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:14:01.244896    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh" event={"ID":"2856e2ae-2d5a-4936-a71a-1b9047b708a9","Type":"ContainerStarted","Data":"efc8148fafe87e71458946a020c1e8b080ec579598e09804d5ee8acc5cbed64b"}
Jul 02 14:14:01.270258 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:14:01.270188    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh" podStartSLOduration=2.7027037910000002 podStartE2EDuration="7.270166949s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.567463158s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:14:01.269776286 +0000 UTC m=+3552.497687465" watchObservedRunningTime="2026-07-02 14:14:01.270166949 +0000 UTC m=+3552.498078123"
Jul 02 14:14:02.013689 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:14:02.013647    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"
Jul 02 14:14:05.013196 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:14:05.013099    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"
Jul 02 14:14:05.014699 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:14:05.014669    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh" podUID="2856e2ae-2d5a-4936-a71a-1b9047b708a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 02 14:14:15.013977 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:14:15.013917    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh" podUID="2856e2ae-2d5a-4936-a71a-1b9047b708a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 02 14:14:25.013490 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:14:25.013440    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh" podUID="2856e2ae-2d5a-4936-a71a-1b9047b708a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 02 14:14:27.403546 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:14:27.403508    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:14:29.234283 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:14:29.234244    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9"
Jul 02 14:14:35.014055 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:14:35.014012    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh" podUID="2856e2ae-2d5a-4936-a71a-1b9047b708a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 02 14:14:43.080322 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:14:43.080202    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:14:45.013805 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:14:45.013763    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh" podUID="2856e2ae-2d5a-4936-a71a-1b9047b708a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 02 14:14:49.562532 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:14:49.562497    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 14:14:49.573558 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:14:49.573530    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 14:14:55.013815 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:14:55.013767    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh" podUID="2856e2ae-2d5a-4936-a71a-1b9047b708a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 02 14:15:05.013990 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:05.013871    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh" podUID="2856e2ae-2d5a-4936-a71a-1b9047b708a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 02 14:15:15.013957 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:15.013913    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh" podUID="2856e2ae-2d5a-4936-a71a-1b9047b708a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 02 14:15:25.013642 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:25.013586    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh" podUID="2856e2ae-2d5a-4936-a71a-1b9047b708a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 02 14:15:29.387015 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:29.386977    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:15:29.405644 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:29.405608    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48"]
Jul 02 14:15:29.415176 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:15:29.415134    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 02 14:15:29.415176 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 14:15:29.415176 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 14:15:29.415176 ip-10-0-135-143 kubenswrapper[2583]:  > containerID="864d4391b3aa35b644fd06eda60fc2901fad852d9c7359aa620c4c282b649016" cmd=["/bin/sleep","15"]
Jul 02 14:15:29.415483 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:29.415189    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48" message="" err=<
Jul 02 14:15:29.415483 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 14:15:29.415483 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 14:15:29.415483 ip-10-0-135-143 kubenswrapper[2583]:  >
Jul 02 14:15:29.415483 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:15:29.415201    2583 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 02 14:15:29.415483 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 14:15:29.415483 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 14:15:29.415483 ip-10-0-135-143 kubenswrapper[2583]:  > pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48" podUID="7c3df67b-64f5-4a86-af39-c1f670ccc437" containerName="main" containerID="cri-o://864d4391b3aa35b644fd06eda60fc2901fad852d9c7359aa620c4c282b649016"
Jul 02 14:15:29.415483 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:29.415248    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48" podUID="7c3df67b-64f5-4a86-af39-c1f670ccc437" containerName="main" containerID="cri-o://864d4391b3aa35b644fd06eda60fc2901fad852d9c7359aa620c4c282b649016" gracePeriod=60
Jul 02 14:15:29.479124 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:29.479085    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:15:29.667079 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:29.667044    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48"
Jul 02 14:15:29.735664 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:29.735632    2583 generic.go:356] "Generic (PLEG): container finished" podID="7c3df67b-64f5-4a86-af39-c1f670ccc437" containerID="864d4391b3aa35b644fd06eda60fc2901fad852d9c7359aa620c4c282b649016" exitCode=0
Jul 02 14:15:29.735865 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:29.735688    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48"
Jul 02 14:15:29.735865 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:29.735712    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48" event={"ID":"7c3df67b-64f5-4a86-af39-c1f670ccc437","Type":"ContainerDied","Data":"864d4391b3aa35b644fd06eda60fc2901fad852d9c7359aa620c4c282b649016"}
Jul 02 14:15:29.735865 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:29.735773    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48" event={"ID":"7c3df67b-64f5-4a86-af39-c1f670ccc437","Type":"ContainerDied","Data":"e1c178ffe2947abf041893423e8c0b5c84a89187c1aacccb057a6647c9ca9247"}
Jul 02 14:15:29.735865 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:29.735802    2583 scope.go:117] "RemoveContainer" containerID="864d4391b3aa35b644fd06eda60fc2901fad852d9c7359aa620c4c282b649016"
Jul 02 14:15:29.745252 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:29.745229    2583 scope.go:117] "RemoveContainer" containerID="a92b1f172b4910bc215040f2062be99e6a3e3575127e2a444bcea9747694b5b6"
Jul 02 14:15:29.754466 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:29.754445    2583 scope.go:117] "RemoveContainer" containerID="864d4391b3aa35b644fd06eda60fc2901fad852d9c7359aa620c4c282b649016"
Jul 02 14:15:29.754754 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:15:29.754735    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"864d4391b3aa35b644fd06eda60fc2901fad852d9c7359aa620c4c282b649016\": container with ID starting with 864d4391b3aa35b644fd06eda60fc2901fad852d9c7359aa620c4c282b649016 not found: ID does not exist" containerID="864d4391b3aa35b644fd06eda60fc2901fad852d9c7359aa620c4c282b649016"
Jul 02 14:15:29.754833 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:29.754768    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"864d4391b3aa35b644fd06eda60fc2901fad852d9c7359aa620c4c282b649016"} err="failed to get container status \"864d4391b3aa35b644fd06eda60fc2901fad852d9c7359aa620c4c282b649016\": rpc error: code = NotFound desc = could not find container \"864d4391b3aa35b644fd06eda60fc2901fad852d9c7359aa620c4c282b649016\": container with ID starting with 864d4391b3aa35b644fd06eda60fc2901fad852d9c7359aa620c4c282b649016 not found: ID does not exist"
Jul 02 14:15:29.754833 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:29.754793    2583 scope.go:117] "RemoveContainer" containerID="a92b1f172b4910bc215040f2062be99e6a3e3575127e2a444bcea9747694b5b6"
Jul 02 14:15:29.755044 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:15:29.755027    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a92b1f172b4910bc215040f2062be99e6a3e3575127e2a444bcea9747694b5b6\": container with ID starting with a92b1f172b4910bc215040f2062be99e6a3e3575127e2a444bcea9747694b5b6 not found: ID does not exist" containerID="a92b1f172b4910bc215040f2062be99e6a3e3575127e2a444bcea9747694b5b6"
Jul 02 14:15:29.755089 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:29.755051    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a92b1f172b4910bc215040f2062be99e6a3e3575127e2a444bcea9747694b5b6"} err="failed to get container status \"a92b1f172b4910bc215040f2062be99e6a3e3575127e2a444bcea9747694b5b6\": rpc error: code = NotFound desc = could not find container \"a92b1f172b4910bc215040f2062be99e6a3e3575127e2a444bcea9747694b5b6\": container with ID starting with a92b1f172b4910bc215040f2062be99e6a3e3575127e2a444bcea9747694b5b6 not found: ID does not exist"
Jul 02 14:15:29.803577 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:29.803543    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djf6b\" (UniqueName: \"kubernetes.io/projected/7c3df67b-64f5-4a86-af39-c1f670ccc437-kube-api-access-djf6b\") pod \"7c3df67b-64f5-4a86-af39-c1f670ccc437\" (UID: \"7c3df67b-64f5-4a86-af39-c1f670ccc437\") "
Jul 02 14:15:29.803748 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:29.803675    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7c3df67b-64f5-4a86-af39-c1f670ccc437-tokenizer-uds\") pod \"7c3df67b-64f5-4a86-af39-c1f670ccc437\" (UID: \"7c3df67b-64f5-4a86-af39-c1f670ccc437\") "
Jul 02 14:15:29.803748 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:29.803745    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c3df67b-64f5-4a86-af39-c1f670ccc437-tls-certs\") pod \"7c3df67b-64f5-4a86-af39-c1f670ccc437\" (UID: \"7c3df67b-64f5-4a86-af39-c1f670ccc437\") "
Jul 02 14:15:29.803867 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:29.803794    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c3df67b-64f5-4a86-af39-c1f670ccc437-kserve-provision-location\") pod \"7c3df67b-64f5-4a86-af39-c1f670ccc437\" (UID: \"7c3df67b-64f5-4a86-af39-c1f670ccc437\") "
Jul 02 14:15:29.804006 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:29.803982    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3df67b-64f5-4a86-af39-c1f670ccc437-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "7c3df67b-64f5-4a86-af39-c1f670ccc437" (UID: "7c3df67b-64f5-4a86-af39-c1f670ccc437"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:15:29.804105 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:29.804086    2583 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7c3df67b-64f5-4a86-af39-c1f670ccc437-tokenizer-uds\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:15:29.804343 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:29.804318    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3df67b-64f5-4a86-af39-c1f670ccc437-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7c3df67b-64f5-4a86-af39-c1f670ccc437" (UID: "7c3df67b-64f5-4a86-af39-c1f670ccc437"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:15:29.805802 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:29.805771    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c3df67b-64f5-4a86-af39-c1f670ccc437-kube-api-access-djf6b" (OuterVolumeSpecName: "kube-api-access-djf6b") pod "7c3df67b-64f5-4a86-af39-c1f670ccc437" (UID: "7c3df67b-64f5-4a86-af39-c1f670ccc437"). InnerVolumeSpecName "kube-api-access-djf6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:15:29.805896 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:29.805806    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c3df67b-64f5-4a86-af39-c1f670ccc437-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c3df67b-64f5-4a86-af39-c1f670ccc437" (UID: "7c3df67b-64f5-4a86-af39-c1f670ccc437"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:15:29.905574 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:29.905481    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c3df67b-64f5-4a86-af39-c1f670ccc437-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:15:29.905574 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:29.905522    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c3df67b-64f5-4a86-af39-c1f670ccc437-kserve-provision-location\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:15:29.905574 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:29.905531    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djf6b\" (UniqueName: \"kubernetes.io/projected/7c3df67b-64f5-4a86-af39-c1f670ccc437-kube-api-access-djf6b\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:15:30.061533 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:30.061487    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48"]
Jul 02 14:15:30.067608 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:30.067578    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4dx48"]
Jul 02 14:15:31.233530 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:31.233489    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c3df67b-64f5-4a86-af39-c1f670ccc437" path="/var/lib/kubelet/pods/7c3df67b-64f5-4a86-af39-c1f670ccc437/volumes"
Jul 02 14:15:35.014116 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:35.014063    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh" podUID="2856e2ae-2d5a-4936-a71a-1b9047b708a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 02 14:15:45.013809 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:45.013752    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh" podUID="2856e2ae-2d5a-4936-a71a-1b9047b708a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 02 14:15:45.468418 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:45.468373    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 02 14:15:55.013642 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:15:55.013587    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh" podUID="2856e2ae-2d5a-4936-a71a-1b9047b708a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 02 14:16:00.492807 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:00.492740    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="541a01ad-2163-47c8-9962-4c77d906de02" containerName="main" containerID="cri-o://f1b41d5994c8256fe3193ff360fa643e32aef1c8d0c630c861c7b0ddcb93557e" gracePeriod=45
Jul 02 14:16:01.252694 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.252662    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 02 14:16:01.318534 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.318437    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/541a01ad-2163-47c8-9962-4c77d906de02-dshm\") pod \"541a01ad-2163-47c8-9962-4c77d906de02\" (UID: \"541a01ad-2163-47c8-9962-4c77d906de02\") "
Jul 02 14:16:01.318534 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.318511    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/541a01ad-2163-47c8-9962-4c77d906de02-tmp-dir\") pod \"541a01ad-2163-47c8-9962-4c77d906de02\" (UID: \"541a01ad-2163-47c8-9962-4c77d906de02\") "
Jul 02 14:16:01.318765 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.318551    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2grjb\" (UniqueName: \"kubernetes.io/projected/541a01ad-2163-47c8-9962-4c77d906de02-kube-api-access-2grjb\") pod \"541a01ad-2163-47c8-9962-4c77d906de02\" (UID: \"541a01ad-2163-47c8-9962-4c77d906de02\") "
Jul 02 14:16:01.318765 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.318580    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/541a01ad-2163-47c8-9962-4c77d906de02-tls-certs\") pod \"541a01ad-2163-47c8-9962-4c77d906de02\" (UID: \"541a01ad-2163-47c8-9962-4c77d906de02\") "
Jul 02 14:16:01.318765 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.318608    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/541a01ad-2163-47c8-9962-4c77d906de02-home\") pod \"541a01ad-2163-47c8-9962-4c77d906de02\" (UID: \"541a01ad-2163-47c8-9962-4c77d906de02\") "
Jul 02 14:16:01.318765 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.318750    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/541a01ad-2163-47c8-9962-4c77d906de02-kserve-provision-location\") pod \"541a01ad-2163-47c8-9962-4c77d906de02\" (UID: \"541a01ad-2163-47c8-9962-4c77d906de02\") "
Jul 02 14:16:01.318990 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.318817    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/541a01ad-2163-47c8-9962-4c77d906de02-model-cache\") pod \"541a01ad-2163-47c8-9962-4c77d906de02\" (UID: \"541a01ad-2163-47c8-9962-4c77d906de02\") "
Jul 02 14:16:01.319296 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.319271    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/541a01ad-2163-47c8-9962-4c77d906de02-home" (OuterVolumeSpecName: "home") pod "541a01ad-2163-47c8-9962-4c77d906de02" (UID: "541a01ad-2163-47c8-9962-4c77d906de02"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:16:01.319872 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.319846    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/541a01ad-2163-47c8-9962-4c77d906de02-model-cache" (OuterVolumeSpecName: "model-cache") pod "541a01ad-2163-47c8-9962-4c77d906de02" (UID: "541a01ad-2163-47c8-9962-4c77d906de02"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:16:01.320800 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.320779    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/541a01ad-2163-47c8-9962-4c77d906de02-dshm" (OuterVolumeSpecName: "dshm") pod "541a01ad-2163-47c8-9962-4c77d906de02" (UID: "541a01ad-2163-47c8-9962-4c77d906de02"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:16:01.321207 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.321179    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/541a01ad-2163-47c8-9962-4c77d906de02-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "541a01ad-2163-47c8-9962-4c77d906de02" (UID: "541a01ad-2163-47c8-9962-4c77d906de02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:16:01.321287 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.321190    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/541a01ad-2163-47c8-9962-4c77d906de02-kube-api-access-2grjb" (OuterVolumeSpecName: "kube-api-access-2grjb") pod "541a01ad-2163-47c8-9962-4c77d906de02" (UID: "541a01ad-2163-47c8-9962-4c77d906de02"). InnerVolumeSpecName "kube-api-access-2grjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:16:01.329800 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.329771    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/541a01ad-2163-47c8-9962-4c77d906de02-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "541a01ad-2163-47c8-9962-4c77d906de02" (UID: "541a01ad-2163-47c8-9962-4c77d906de02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:16:01.364632 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.364585    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/541a01ad-2163-47c8-9962-4c77d906de02-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "541a01ad-2163-47c8-9962-4c77d906de02" (UID: "541a01ad-2163-47c8-9962-4c77d906de02"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:16:01.420501 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.420458    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/541a01ad-2163-47c8-9962-4c77d906de02-tmp-dir\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:16:01.420501 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.420494    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2grjb\" (UniqueName: \"kubernetes.io/projected/541a01ad-2163-47c8-9962-4c77d906de02-kube-api-access-2grjb\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:16:01.420501 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.420507    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/541a01ad-2163-47c8-9962-4c77d906de02-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:16:01.420748 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.420518    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/541a01ad-2163-47c8-9962-4c77d906de02-home\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:16:01.420748 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.420529    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/541a01ad-2163-47c8-9962-4c77d906de02-kserve-provision-location\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:16:01.420748 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.420541    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/541a01ad-2163-47c8-9962-4c77d906de02-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:16:01.420748 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.420553    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/541a01ad-2163-47c8-9962-4c77d906de02-dshm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:16:01.910688 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.910655    2583 generic.go:356] "Generic (PLEG): container finished" podID="541a01ad-2163-47c8-9962-4c77d906de02" containerID="f1b41d5994c8256fe3193ff360fa643e32aef1c8d0c630c861c7b0ddcb93557e" exitCode=0
Jul 02 14:16:01.911057 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.910711    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 02 14:16:01.911057 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.910738    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"541a01ad-2163-47c8-9962-4c77d906de02","Type":"ContainerDied","Data":"f1b41d5994c8256fe3193ff360fa643e32aef1c8d0c630c861c7b0ddcb93557e"}
Jul 02 14:16:01.911057 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.910792    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"541a01ad-2163-47c8-9962-4c77d906de02","Type":"ContainerDied","Data":"011502b23bd843a2d3b65bac8715cdb6bce072d80f44e634a19514c0666b7ef4"}
Jul 02 14:16:01.911057 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.910814    2583 scope.go:117] "RemoveContainer" containerID="f1b41d5994c8256fe3193ff360fa643e32aef1c8d0c630c861c7b0ddcb93557e"
Jul 02 14:16:01.920330 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.920309    2583 scope.go:117] "RemoveContainer" containerID="0cd1df5272fbb9a96504b31c81780130fd7f85a12e4f2b9cd2008f5dd78b50b9"
Jul 02 14:16:01.929173 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.929149    2583 scope.go:117] "RemoveContainer" containerID="f1b41d5994c8256fe3193ff360fa643e32aef1c8d0c630c861c7b0ddcb93557e"
Jul 02 14:16:01.929510 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:16:01.929482    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1b41d5994c8256fe3193ff360fa643e32aef1c8d0c630c861c7b0ddcb93557e\": container with ID starting with f1b41d5994c8256fe3193ff360fa643e32aef1c8d0c630c861c7b0ddcb93557e not found: ID does not exist" containerID="f1b41d5994c8256fe3193ff360fa643e32aef1c8d0c630c861c7b0ddcb93557e"
Jul 02 14:16:01.929607 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.929523    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1b41d5994c8256fe3193ff360fa643e32aef1c8d0c630c861c7b0ddcb93557e"} err="failed to get container status \"f1b41d5994c8256fe3193ff360fa643e32aef1c8d0c630c861c7b0ddcb93557e\": rpc error: code = NotFound desc = could not find container \"f1b41d5994c8256fe3193ff360fa643e32aef1c8d0c630c861c7b0ddcb93557e\": container with ID starting with f1b41d5994c8256fe3193ff360fa643e32aef1c8d0c630c861c7b0ddcb93557e not found: ID does not exist"
Jul 02 14:16:01.929607 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.929547    2583 scope.go:117] "RemoveContainer" containerID="0cd1df5272fbb9a96504b31c81780130fd7f85a12e4f2b9cd2008f5dd78b50b9"
Jul 02 14:16:01.929817 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:16:01.929799    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cd1df5272fbb9a96504b31c81780130fd7f85a12e4f2b9cd2008f5dd78b50b9\": container with ID starting with 0cd1df5272fbb9a96504b31c81780130fd7f85a12e4f2b9cd2008f5dd78b50b9 not found: ID does not exist" containerID="0cd1df5272fbb9a96504b31c81780130fd7f85a12e4f2b9cd2008f5dd78b50b9"
Jul 02 14:16:01.929857 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.929824    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cd1df5272fbb9a96504b31c81780130fd7f85a12e4f2b9cd2008f5dd78b50b9"} err="failed to get container status \"0cd1df5272fbb9a96504b31c81780130fd7f85a12e4f2b9cd2008f5dd78b50b9\": rpc error: code = NotFound desc = could not find container \"0cd1df5272fbb9a96504b31c81780130fd7f85a12e4f2b9cd2008f5dd78b50b9\": container with ID starting with 0cd1df5272fbb9a96504b31c81780130fd7f85a12e4f2b9cd2008f5dd78b50b9 not found: ID does not exist"
Jul 02 14:16:01.934485 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.934445    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 02 14:16:01.938998 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:01.938967    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 02 14:16:03.234589 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:03.234546    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="541a01ad-2163-47c8-9962-4c77d906de02" path="/var/lib/kubelet/pods/541a01ad-2163-47c8-9962-4c77d906de02/volumes"
Jul 02 14:16:05.023964 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:05.023874    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"
Jul 02 14:16:05.032307 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:05.032279    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"
Jul 02 14:16:33.700782 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:33.700700    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c"]
Jul 02 14:16:33.701220 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:33.701092    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c3df67b-64f5-4a86-af39-c1f670ccc437" containerName="storage-initializer"
Jul 02 14:16:33.701220 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:33.701103    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c3df67b-64f5-4a86-af39-c1f670ccc437" containerName="storage-initializer"
Jul 02 14:16:33.701220 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:33.701119    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="541a01ad-2163-47c8-9962-4c77d906de02" containerName="storage-initializer"
Jul 02 14:16:33.701220 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:33.701124    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="541a01ad-2163-47c8-9962-4c77d906de02" containerName="storage-initializer"
Jul 02 14:16:33.701220 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:33.701141    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c3df67b-64f5-4a86-af39-c1f670ccc437" containerName="main"
Jul 02 14:16:33.701220 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:33.701147    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c3df67b-64f5-4a86-af39-c1f670ccc437" containerName="main"
Jul 02 14:16:33.701220 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:33.701155    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="541a01ad-2163-47c8-9962-4c77d906de02" containerName="main"
Jul 02 14:16:33.701220 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:33.701160    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="541a01ad-2163-47c8-9962-4c77d906de02" containerName="main"
Jul 02 14:16:33.701220 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:33.701212    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c3df67b-64f5-4a86-af39-c1f670ccc437" containerName="main"
Jul 02 14:16:33.701220 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:33.701224    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="541a01ad-2163-47c8-9962-4c77d906de02" containerName="main"
Jul 02 14:16:33.704504 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:33.704478    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c"
Jul 02 14:16:33.707181 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:33.707160    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 02 14:16:33.716201 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:33.716173    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c"]
Jul 02 14:16:33.722446 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:33.722417    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-dshm\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c\" (UID: \"ba8f91e7-ae35-48af-b3ce-254f00abc9dd\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c"
Jul 02 14:16:33.722446 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:33.722447    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr6kn\" (UniqueName: \"kubernetes.io/projected/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-kube-api-access-cr6kn\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c\" (UID: \"ba8f91e7-ae35-48af-b3ce-254f00abc9dd\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c"
Jul 02 14:16:33.722665 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:33.722535    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-tls-certs\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c\" (UID: \"ba8f91e7-ae35-48af-b3ce-254f00abc9dd\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c"
Jul 02 14:16:33.722665 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:33.722566    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-home\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c\" (UID: \"ba8f91e7-ae35-48af-b3ce-254f00abc9dd\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c"
Jul 02 14:16:33.722665 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:33.722588    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-model-cache\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c\" (UID: \"ba8f91e7-ae35-48af-b3ce-254f00abc9dd\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c"
Jul 02 14:16:33.722665 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:33.722604    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-tmp-dir\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c\" (UID: \"ba8f91e7-ae35-48af-b3ce-254f00abc9dd\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c"
Jul 02 14:16:33.738857 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:33.738824    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:16:33.823758 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:33.823718    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-tls-certs\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c\" (UID: \"ba8f91e7-ae35-48af-b3ce-254f00abc9dd\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c"
Jul 02 14:16:33.823957 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:33.823767    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-home\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c\" (UID: \"ba8f91e7-ae35-48af-b3ce-254f00abc9dd\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c"
Jul 02 14:16:33.823957 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:33.823794    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-model-cache\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c\" (UID: \"ba8f91e7-ae35-48af-b3ce-254f00abc9dd\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c"
Jul 02 14:16:33.823957 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:33.823910    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-tmp-dir\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c\" (UID: \"ba8f91e7-ae35-48af-b3ce-254f00abc9dd\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c"
Jul 02 14:16:33.824141 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:33.824003    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-dshm\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c\" (UID: \"ba8f91e7-ae35-48af-b3ce-254f00abc9dd\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c"
Jul 02 14:16:33.824141 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:33.824033    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr6kn\" (UniqueName: \"kubernetes.io/projected/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-kube-api-access-cr6kn\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c\" (UID: \"ba8f91e7-ae35-48af-b3ce-254f00abc9dd\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c"
Jul 02 14:16:33.824245 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:33.824177    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-model-cache\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c\" (UID: \"ba8f91e7-ae35-48af-b3ce-254f00abc9dd\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c"
Jul 02 14:16:33.824245 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:33.824220    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-home\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c\" (UID: \"ba8f91e7-ae35-48af-b3ce-254f00abc9dd\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c"
Jul 02 14:16:33.824245 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:33.824242    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-tmp-dir\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c\" (UID: \"ba8f91e7-ae35-48af-b3ce-254f00abc9dd\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c"
Jul 02 14:16:33.826265 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:33.826241    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-dshm\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c\" (UID: \"ba8f91e7-ae35-48af-b3ce-254f00abc9dd\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c"
Jul 02 14:16:33.826462 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:33.826445    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-tls-certs\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c\" (UID: \"ba8f91e7-ae35-48af-b3ce-254f00abc9dd\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c"
Jul 02 14:16:33.833997 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:33.833974    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr6kn\" (UniqueName: \"kubernetes.io/projected/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-kube-api-access-cr6kn\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c\" (UID: \"ba8f91e7-ae35-48af-b3ce-254f00abc9dd\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c"
Jul 02 14:16:34.015939 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:34.015847    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c"
Jul 02 14:16:34.155250 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:34.155221    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c"]
Jul 02 14:16:34.157378 ip-10-0-135-143 kubenswrapper[2583]: W0702 14:16:34.157349    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba8f91e7_ae35_48af_b3ce_254f00abc9dd.slice/crio-ab96e739924a2a0bd180001663da296da1b0eec0a783edbce1951c92a63abe8f WatchSource:0}: Error finding container ab96e739924a2a0bd180001663da296da1b0eec0a783edbce1951c92a63abe8f: Status 404 returned error can't find the container with id ab96e739924a2a0bd180001663da296da1b0eec0a783edbce1951c92a63abe8f
Jul 02 14:16:34.272613 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:34.272528    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:16:35.084890 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:35.084853    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c" event={"ID":"ba8f91e7-ae35-48af-b3ce-254f00abc9dd","Type":"ContainerStarted","Data":"4ea9567789063454b12ec765c398d540f166dd0505981d2082be2be628c7080a"}
Jul 02 14:16:35.085320 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:35.084912    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c" event={"ID":"ba8f91e7-ae35-48af-b3ce-254f00abc9dd","Type":"ContainerStarted","Data":"ab96e739924a2a0bd180001663da296da1b0eec0a783edbce1951c92a63abe8f"}
Jul 02 14:16:35.109951 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:35.109892    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c" podStartSLOduration=2.109874547 podStartE2EDuration="2.109874547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:16:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:16:35.108885629 +0000 UTC m=+3706.336796804" watchObservedRunningTime="2026-07-02 14:16:35.109874547 +0000 UTC m=+3706.337785722"
Jul 02 14:16:36.016907 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:36.016870    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c"
Jul 02 14:16:44.016508 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:44.016463    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c"
Jul 02 14:16:44.033225 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:44.033192    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c"
Jul 02 14:16:44.144383 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:16:44.144351    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c"
Jul 02 14:17:10.921715 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:10.921670    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"]
Jul 02 14:17:10.954498 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:10.954462    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:17:11.006873 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:11.006840    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9"]
Jul 02 14:17:11.017436 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:17:11.017376    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 02 14:17:11.017436 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 14:17:11.017436 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 14:17:11.017436 ip-10-0-135-143 kubenswrapper[2583]:  > containerID="f57d850ec97fabc6fbfa4acc76bd2684375f0e6f34ccd0f98c48716f70d97cf3" cmd=["/bin/sleep","15"]
Jul 02 14:17:11.017714 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:11.017452    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9" message="" err=<
Jul 02 14:17:11.017714 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 14:17:11.017714 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 14:17:11.017714 ip-10-0-135-143 kubenswrapper[2583]:  >
Jul 02 14:17:11.017714 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:17:11.017465    2583 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 02 14:17:11.017714 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 14:17:11.017714 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 14:17:11.017714 ip-10-0-135-143 kubenswrapper[2583]:  > pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9" podUID="99157bb7-73e8-466f-ad6a-a799dffee0a2" containerName="main" containerID="cri-o://f57d850ec97fabc6fbfa4acc76bd2684375f0e6f34ccd0f98c48716f70d97cf3"
Jul 02 14:17:11.017714 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:11.017522    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9" podUID="99157bb7-73e8-466f-ad6a-a799dffee0a2" containerName="main" containerID="cri-o://f57d850ec97fabc6fbfa4acc76bd2684375f0e6f34ccd0f98c48716f70d97cf3" gracePeriod=60
Jul 02 14:17:11.111074 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:11.111044    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:17:11.212866 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:11.212816    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:17:11.274157 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:11.274126    2583 generic.go:356] "Generic (PLEG): container finished" podID="99157bb7-73e8-466f-ad6a-a799dffee0a2" containerID="f57d850ec97fabc6fbfa4acc76bd2684375f0e6f34ccd0f98c48716f70d97cf3" exitCode=0
Jul 02 14:17:11.274329 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:11.274185    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9" event={"ID":"99157bb7-73e8-466f-ad6a-a799dffee0a2","Type":"ContainerDied","Data":"f57d850ec97fabc6fbfa4acc76bd2684375f0e6f34ccd0f98c48716f70d97cf3"}
Jul 02 14:17:11.274329 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:11.274220    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9" event={"ID":"99157bb7-73e8-466f-ad6a-a799dffee0a2","Type":"ContainerDied","Data":"2534cad8c9111d87e8e4b039a91cda26af553797ae8bd9fabb29987cd3a824c7"}
Jul 02 14:17:11.274329 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:11.274235    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2534cad8c9111d87e8e4b039a91cda26af553797ae8bd9fabb29987cd3a824c7"
Jul 02 14:17:11.281139 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:11.281117    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9"
Jul 02 14:17:11.389039 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:11.389007    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2nqz\" (UniqueName: \"kubernetes.io/projected/99157bb7-73e8-466f-ad6a-a799dffee0a2-kube-api-access-n2nqz\") pod \"99157bb7-73e8-466f-ad6a-a799dffee0a2\" (UID: \"99157bb7-73e8-466f-ad6a-a799dffee0a2\") "
Jul 02 14:17:11.389223 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:11.389058    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99157bb7-73e8-466f-ad6a-a799dffee0a2-kserve-provision-location\") pod \"99157bb7-73e8-466f-ad6a-a799dffee0a2\" (UID: \"99157bb7-73e8-466f-ad6a-a799dffee0a2\") "
Jul 02 14:17:11.389223 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:11.389096    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/99157bb7-73e8-466f-ad6a-a799dffee0a2-tokenizer-uds\") pod \"99157bb7-73e8-466f-ad6a-a799dffee0a2\" (UID: \"99157bb7-73e8-466f-ad6a-a799dffee0a2\") "
Jul 02 14:17:11.389223 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:11.389145    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99157bb7-73e8-466f-ad6a-a799dffee0a2-tls-certs\") pod \"99157bb7-73e8-466f-ad6a-a799dffee0a2\" (UID: \"99157bb7-73e8-466f-ad6a-a799dffee0a2\") "
Jul 02 14:17:11.389441 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:11.389412    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99157bb7-73e8-466f-ad6a-a799dffee0a2-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "99157bb7-73e8-466f-ad6a-a799dffee0a2" (UID: "99157bb7-73e8-466f-ad6a-a799dffee0a2"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:17:11.389703 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:11.389682    2583 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/99157bb7-73e8-466f-ad6a-a799dffee0a2-tokenizer-uds\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:17:11.389816 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:11.389790    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99157bb7-73e8-466f-ad6a-a799dffee0a2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "99157bb7-73e8-466f-ad6a-a799dffee0a2" (UID: "99157bb7-73e8-466f-ad6a-a799dffee0a2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:17:11.391245 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:11.391222    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99157bb7-73e8-466f-ad6a-a799dffee0a2-kube-api-access-n2nqz" (OuterVolumeSpecName: "kube-api-access-n2nqz") pod "99157bb7-73e8-466f-ad6a-a799dffee0a2" (UID: "99157bb7-73e8-466f-ad6a-a799dffee0a2"). InnerVolumeSpecName "kube-api-access-n2nqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:17:11.391245 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:11.391229    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99157bb7-73e8-466f-ad6a-a799dffee0a2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "99157bb7-73e8-466f-ad6a-a799dffee0a2" (UID: "99157bb7-73e8-466f-ad6a-a799dffee0a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:17:11.491043 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:11.490959    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99157bb7-73e8-466f-ad6a-a799dffee0a2-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:17:11.491043 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:11.490987    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2nqz\" (UniqueName: \"kubernetes.io/projected/99157bb7-73e8-466f-ad6a-a799dffee0a2-kube-api-access-n2nqz\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:17:11.491043 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:11.490999    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99157bb7-73e8-466f-ad6a-a799dffee0a2-kserve-provision-location\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:17:11.937015 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:11.936976    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:17:12.057199 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:12.057167    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:17:12.278959 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:12.278876    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9"
Jul 02 14:17:12.313724 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:12.313692    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9"]
Jul 02 14:17:12.323484 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:12.323450    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-krbz9"]
Jul 02 14:17:13.238750 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:13.238718    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99157bb7-73e8-466f-ad6a-a799dffee0a2" path="/var/lib/kubelet/pods/99157bb7-73e8-466f-ad6a-a799dffee0a2/volumes"
Jul 02 14:17:25.945512 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:25.945455    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh" podUID="2856e2ae-2d5a-4936-a71a-1b9047b708a9" containerName="main" containerID="cri-o://efc8148fafe87e71458946a020c1e8b080ec579598e09804d5ee8acc5cbed64b" gracePeriod=45
Jul 02 14:17:26.025295 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:26.025246    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh" podUID="2856e2ae-2d5a-4936-a71a-1b9047b708a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 02 14:17:26.713566 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:26.713537    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"
Jul 02 14:17:26.838761 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:26.838678    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2856e2ae-2d5a-4936-a71a-1b9047b708a9-kserve-provision-location\") pod \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\" (UID: \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\") "
Jul 02 14:17:26.838761 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:26.838729    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2856e2ae-2d5a-4936-a71a-1b9047b708a9-model-cache\") pod \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\" (UID: \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\") "
Jul 02 14:17:26.838989 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:26.838770    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2856e2ae-2d5a-4936-a71a-1b9047b708a9-home\") pod \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\" (UID: \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\") "
Jul 02 14:17:26.838989 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:26.838796    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgqc9\" (UniqueName: \"kubernetes.io/projected/2856e2ae-2d5a-4936-a71a-1b9047b708a9-kube-api-access-xgqc9\") pod \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\" (UID: \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\") "
Jul 02 14:17:26.838989 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:26.838815    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2856e2ae-2d5a-4936-a71a-1b9047b708a9-tmp-dir\") pod \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\" (UID: \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\") "
Jul 02 14:17:26.838989 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:26.838830    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2856e2ae-2d5a-4936-a71a-1b9047b708a9-tls-certs\") pod \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\" (UID: \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\") "
Jul 02 14:17:26.838989 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:26.838854    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2856e2ae-2d5a-4936-a71a-1b9047b708a9-dshm\") pod \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\" (UID: \"2856e2ae-2d5a-4936-a71a-1b9047b708a9\") "
Jul 02 14:17:26.839259 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:26.839117    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2856e2ae-2d5a-4936-a71a-1b9047b708a9-model-cache" (OuterVolumeSpecName: "model-cache") pod "2856e2ae-2d5a-4936-a71a-1b9047b708a9" (UID: "2856e2ae-2d5a-4936-a71a-1b9047b708a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:17:26.839316 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:26.839273    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2856e2ae-2d5a-4936-a71a-1b9047b708a9-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:17:26.839518 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:26.839493    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2856e2ae-2d5a-4936-a71a-1b9047b708a9-home" (OuterVolumeSpecName: "home") pod "2856e2ae-2d5a-4936-a71a-1b9047b708a9" (UID: "2856e2ae-2d5a-4936-a71a-1b9047b708a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:17:26.841144 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:26.841111    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2856e2ae-2d5a-4936-a71a-1b9047b708a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2856e2ae-2d5a-4936-a71a-1b9047b708a9" (UID: "2856e2ae-2d5a-4936-a71a-1b9047b708a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:17:26.841513 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:26.841380    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2856e2ae-2d5a-4936-a71a-1b9047b708a9-dshm" (OuterVolumeSpecName: "dshm") pod "2856e2ae-2d5a-4936-a71a-1b9047b708a9" (UID: "2856e2ae-2d5a-4936-a71a-1b9047b708a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:17:26.841513 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:26.841445    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2856e2ae-2d5a-4936-a71a-1b9047b708a9-kube-api-access-xgqc9" (OuterVolumeSpecName: "kube-api-access-xgqc9") pod "2856e2ae-2d5a-4936-a71a-1b9047b708a9" (UID: "2856e2ae-2d5a-4936-a71a-1b9047b708a9"). InnerVolumeSpecName "kube-api-access-xgqc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:17:26.857766 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:26.857727    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2856e2ae-2d5a-4936-a71a-1b9047b708a9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2856e2ae-2d5a-4936-a71a-1b9047b708a9" (UID: "2856e2ae-2d5a-4936-a71a-1b9047b708a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:17:26.866505 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:26.866473    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2856e2ae-2d5a-4936-a71a-1b9047b708a9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2856e2ae-2d5a-4936-a71a-1b9047b708a9" (UID: "2856e2ae-2d5a-4936-a71a-1b9047b708a9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:17:26.939839 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:26.939790    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgqc9\" (UniqueName: \"kubernetes.io/projected/2856e2ae-2d5a-4936-a71a-1b9047b708a9-kube-api-access-xgqc9\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:17:26.940032 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:26.939860    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2856e2ae-2d5a-4936-a71a-1b9047b708a9-tmp-dir\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:17:26.940032 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:26.939876    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2856e2ae-2d5a-4936-a71a-1b9047b708a9-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:17:26.940032 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:26.939888    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2856e2ae-2d5a-4936-a71a-1b9047b708a9-dshm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:17:26.940032 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:26.939900    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2856e2ae-2d5a-4936-a71a-1b9047b708a9-kserve-provision-location\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:17:26.940032 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:26.939911    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2856e2ae-2d5a-4936-a71a-1b9047b708a9-home\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:17:27.359114 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:27.359077    2583 generic.go:356] "Generic (PLEG): container finished" podID="2856e2ae-2d5a-4936-a71a-1b9047b708a9" containerID="efc8148fafe87e71458946a020c1e8b080ec579598e09804d5ee8acc5cbed64b" exitCode=0
Jul 02 14:17:27.359559 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:27.359131    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"
Jul 02 14:17:27.359559 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:27.359163    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh" event={"ID":"2856e2ae-2d5a-4936-a71a-1b9047b708a9","Type":"ContainerDied","Data":"efc8148fafe87e71458946a020c1e8b080ec579598e09804d5ee8acc5cbed64b"}
Jul 02 14:17:27.359559 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:27.359225    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh" event={"ID":"2856e2ae-2d5a-4936-a71a-1b9047b708a9","Type":"ContainerDied","Data":"80f8786651db9ecff5e08f173cc5f0b784ab71c5e4c2ab93392a745824cabeb6"}
Jul 02 14:17:27.359559 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:27.359256    2583 scope.go:117] "RemoveContainer" containerID="efc8148fafe87e71458946a020c1e8b080ec579598e09804d5ee8acc5cbed64b"
Jul 02 14:17:27.368232 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:27.368213    2583 scope.go:117] "RemoveContainer" containerID="495fb531a9ac43e6c9b855ce5e7b307018cc7ecde4f5c4e52747af239daad572"
Jul 02 14:17:27.371196 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:27.371168    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:17:27.376931 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:27.376907    2583 scope.go:117] "RemoveContainer" containerID="efc8148fafe87e71458946a020c1e8b080ec579598e09804d5ee8acc5cbed64b"
Jul 02 14:17:27.377404 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:17:27.377368    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efc8148fafe87e71458946a020c1e8b080ec579598e09804d5ee8acc5cbed64b\": container with ID starting with efc8148fafe87e71458946a020c1e8b080ec579598e09804d5ee8acc5cbed64b not found: ID does not exist" containerID="efc8148fafe87e71458946a020c1e8b080ec579598e09804d5ee8acc5cbed64b"
Jul 02 14:17:27.377500 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:27.377419    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc8148fafe87e71458946a020c1e8b080ec579598e09804d5ee8acc5cbed64b"} err="failed to get container status \"efc8148fafe87e71458946a020c1e8b080ec579598e09804d5ee8acc5cbed64b\": rpc error: code = NotFound desc = could not find container \"efc8148fafe87e71458946a020c1e8b080ec579598e09804d5ee8acc5cbed64b\": container with ID starting with efc8148fafe87e71458946a020c1e8b080ec579598e09804d5ee8acc5cbed64b not found: ID does not exist"
Jul 02 14:17:27.377500 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:27.377443    2583 scope.go:117] "RemoveContainer" containerID="495fb531a9ac43e6c9b855ce5e7b307018cc7ecde4f5c4e52747af239daad572"
Jul 02 14:17:27.377824 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:17:27.377797    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"495fb531a9ac43e6c9b855ce5e7b307018cc7ecde4f5c4e52747af239daad572\": container with ID starting with 495fb531a9ac43e6c9b855ce5e7b307018cc7ecde4f5c4e52747af239daad572 not found: ID does not exist" containerID="495fb531a9ac43e6c9b855ce5e7b307018cc7ecde4f5c4e52747af239daad572"
Jul 02 14:17:27.377900 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:27.377835    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"495fb531a9ac43e6c9b855ce5e7b307018cc7ecde4f5c4e52747af239daad572"} err="failed to get container status \"495fb531a9ac43e6c9b855ce5e7b307018cc7ecde4f5c4e52747af239daad572\": rpc error: code = NotFound desc = could not find container \"495fb531a9ac43e6c9b855ce5e7b307018cc7ecde4f5c4e52747af239daad572\": container with ID starting with 495fb531a9ac43e6c9b855ce5e7b307018cc7ecde4f5c4e52747af239daad572 not found: ID does not exist"
Jul 02 14:17:27.379756 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:27.379734    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"]
Jul 02 14:17:27.383838 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:27.383808    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-bpbsh"]
Jul 02 14:17:29.233644 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:29.233606    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2856e2ae-2d5a-4936-a71a-1b9047b708a9" path="/var/lib/kubelet/pods/2856e2ae-2d5a-4936-a71a-1b9047b708a9/volumes"
Jul 02 14:17:35.394673 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:35.394638    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:17:35.497254 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:17:35.497217    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:18:07.520099 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.520029    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"]
Jul 02 14:18:07.520615 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.520458    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2856e2ae-2d5a-4936-a71a-1b9047b708a9" containerName="storage-initializer"
Jul 02 14:18:07.520615 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.520478    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2856e2ae-2d5a-4936-a71a-1b9047b708a9" containerName="storage-initializer"
Jul 02 14:18:07.520615 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.520487    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99157bb7-73e8-466f-ad6a-a799dffee0a2" containerName="storage-initializer"
Jul 02 14:18:07.520615 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.520492    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="99157bb7-73e8-466f-ad6a-a799dffee0a2" containerName="storage-initializer"
Jul 02 14:18:07.520615 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.520518    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99157bb7-73e8-466f-ad6a-a799dffee0a2" containerName="main"
Jul 02 14:18:07.520615 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.520526    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="99157bb7-73e8-466f-ad6a-a799dffee0a2" containerName="main"
Jul 02 14:18:07.520615 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.520567    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2856e2ae-2d5a-4936-a71a-1b9047b708a9" containerName="main"
Jul 02 14:18:07.520615 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.520576    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2856e2ae-2d5a-4936-a71a-1b9047b708a9" containerName="main"
Jul 02 14:18:07.520957 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.520679    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="99157bb7-73e8-466f-ad6a-a799dffee0a2" containerName="main"
Jul 02 14:18:07.520957 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.520691    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2856e2ae-2d5a-4936-a71a-1b9047b708a9" containerName="main"
Jul 02 14:18:07.524020 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.523997    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"
Jul 02 14:18:07.527506 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.527480    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 02 14:18:07.528031 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.528012    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 14:18:07.537163 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.537135    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"]
Jul 02 14:18:07.555802 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.555769    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:18:07.621264 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.621224    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-tls-certs\") pod \"stop-feature-test-kserve-757c67794-wrk7x\" (UID: \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"
Jul 02 14:18:07.621264 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.621266    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-tmp-dir\") pod \"stop-feature-test-kserve-757c67794-wrk7x\" (UID: \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"
Jul 02 14:18:07.621557 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.621299    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5gr8\" (UniqueName: \"kubernetes.io/projected/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-kube-api-access-q5gr8\") pod \"stop-feature-test-kserve-757c67794-wrk7x\" (UID: \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"
Jul 02 14:18:07.621557 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.621424    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-home\") pod \"stop-feature-test-kserve-757c67794-wrk7x\" (UID: \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"
Jul 02 14:18:07.621557 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.621454    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-kserve-provision-location\") pod \"stop-feature-test-kserve-757c67794-wrk7x\" (UID: \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"
Jul 02 14:18:07.621670 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.621566    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-dshm\") pod \"stop-feature-test-kserve-757c67794-wrk7x\" (UID: \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"
Jul 02 14:18:07.621670 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.621599    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-model-cache\") pod \"stop-feature-test-kserve-757c67794-wrk7x\" (UID: \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"
Jul 02 14:18:07.722869 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.722832    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-tmp-dir\") pod \"stop-feature-test-kserve-757c67794-wrk7x\" (UID: \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"
Jul 02 14:18:07.723047 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.722876    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5gr8\" (UniqueName: \"kubernetes.io/projected/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-kube-api-access-q5gr8\") pod \"stop-feature-test-kserve-757c67794-wrk7x\" (UID: \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"
Jul 02 14:18:07.723047 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.722996    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-home\") pod \"stop-feature-test-kserve-757c67794-wrk7x\" (UID: \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"
Jul 02 14:18:07.723047 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.723027    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-kserve-provision-location\") pod \"stop-feature-test-kserve-757c67794-wrk7x\" (UID: \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"
Jul 02 14:18:07.723180 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.723110    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-dshm\") pod \"stop-feature-test-kserve-757c67794-wrk7x\" (UID: \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"
Jul 02 14:18:07.723180 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.723149    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-model-cache\") pod \"stop-feature-test-kserve-757c67794-wrk7x\" (UID: \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"
Jul 02 14:18:07.723288 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.723191    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-tls-certs\") pod \"stop-feature-test-kserve-757c67794-wrk7x\" (UID: \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"
Jul 02 14:18:07.723288 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.723239    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-tmp-dir\") pod \"stop-feature-test-kserve-757c67794-wrk7x\" (UID: \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"
Jul 02 14:18:07.723430 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.723371    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-home\") pod \"stop-feature-test-kserve-757c67794-wrk7x\" (UID: \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"
Jul 02 14:18:07.723511 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.723489    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-kserve-provision-location\") pod \"stop-feature-test-kserve-757c67794-wrk7x\" (UID: \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"
Jul 02 14:18:07.723559 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.723506    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-model-cache\") pod \"stop-feature-test-kserve-757c67794-wrk7x\" (UID: \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"
Jul 02 14:18:07.725628 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.725597    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-dshm\") pod \"stop-feature-test-kserve-757c67794-wrk7x\" (UID: \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"
Jul 02 14:18:07.725729 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.725716    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-tls-certs\") pod \"stop-feature-test-kserve-757c67794-wrk7x\" (UID: \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"
Jul 02 14:18:07.732439 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.732378    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5gr8\" (UniqueName: \"kubernetes.io/projected/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-kube-api-access-q5gr8\") pod \"stop-feature-test-kserve-757c67794-wrk7x\" (UID: \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"
Jul 02 14:18:07.835644 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.835555    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"
Jul 02 14:18:07.955021 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.954982    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws"]
Jul 02 14:18:07.959813 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.959787    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws"
Jul 02 14:18:07.970526 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.970506    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-4h4dg\""
Jul 02 14:18:07.981223 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.981189    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws"]
Jul 02 14:18:07.996454 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:07.996430    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"]
Jul 02 14:18:07.998352 ip-10-0-135-143 kubenswrapper[2583]: W0702 14:18:07.998324    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0fac6a5_c73f_4855_a0a6_c8a4b0a3312f.slice/crio-3cf4c440aa12741b58cc8f09ba27361692c3a99808f2c8fa4d9e6dffaa5eb2c4 WatchSource:0}: Error finding container 3cf4c440aa12741b58cc8f09ba27361692c3a99808f2c8fa4d9e6dffaa5eb2c4: Status 404 returned error can't find the container with id 3cf4c440aa12741b58cc8f09ba27361692c3a99808f2c8fa4d9e6dffaa5eb2c4
Jul 02 14:18:08.013205 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:08.013169    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:18:08.026506 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:08.026476    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8a3125bc-aec6-479f-9f15-be02e6406111-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws\" (UID: \"8a3125bc-aec6-479f-9f15-be02e6406111\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws"
Jul 02 14:18:08.026622 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:08.026522    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a3125bc-aec6-479f-9f15-be02e6406111-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws\" (UID: \"8a3125bc-aec6-479f-9f15-be02e6406111\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws"
Jul 02 14:18:08.026622 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:08.026553    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a3125bc-aec6-479f-9f15-be02e6406111-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws\" (UID: \"8a3125bc-aec6-479f-9f15-be02e6406111\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws"
Jul 02 14:18:08.026622 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:08.026599    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c5vn\" (UniqueName: \"kubernetes.io/projected/8a3125bc-aec6-479f-9f15-be02e6406111-kube-api-access-4c5vn\") pod \"stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws\" (UID: \"8a3125bc-aec6-479f-9f15-be02e6406111\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws"
Jul 02 14:18:08.127632 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:08.127590    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c5vn\" (UniqueName: \"kubernetes.io/projected/8a3125bc-aec6-479f-9f15-be02e6406111-kube-api-access-4c5vn\") pod \"stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws\" (UID: \"8a3125bc-aec6-479f-9f15-be02e6406111\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws"
Jul 02 14:18:08.127805 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:08.127707    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8a3125bc-aec6-479f-9f15-be02e6406111-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws\" (UID: \"8a3125bc-aec6-479f-9f15-be02e6406111\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws"
Jul 02 14:18:08.127805 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:08.127738    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a3125bc-aec6-479f-9f15-be02e6406111-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws\" (UID: \"8a3125bc-aec6-479f-9f15-be02e6406111\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws"
Jul 02 14:18:08.127805 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:08.127767    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a3125bc-aec6-479f-9f15-be02e6406111-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws\" (UID: \"8a3125bc-aec6-479f-9f15-be02e6406111\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws"
Jul 02 14:18:08.128104 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:08.128080    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8a3125bc-aec6-479f-9f15-be02e6406111-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws\" (UID: \"8a3125bc-aec6-479f-9f15-be02e6406111\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws"
Jul 02 14:18:08.128182 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:08.128134    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a3125bc-aec6-479f-9f15-be02e6406111-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws\" (UID: \"8a3125bc-aec6-479f-9f15-be02e6406111\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws"
Jul 02 14:18:08.130136 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:08.130116    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a3125bc-aec6-479f-9f15-be02e6406111-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws\" (UID: \"8a3125bc-aec6-479f-9f15-be02e6406111\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws"
Jul 02 14:18:08.136694 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:08.136667    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c5vn\" (UniqueName: \"kubernetes.io/projected/8a3125bc-aec6-479f-9f15-be02e6406111-kube-api-access-4c5vn\") pod \"stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws\" (UID: \"8a3125bc-aec6-479f-9f15-be02e6406111\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws"
Jul 02 14:18:08.270099 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:08.270059    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws"
Jul 02 14:18:08.418669 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:08.418639    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws"]
Jul 02 14:18:08.420854 ip-10-0-135-143 kubenswrapper[2583]: W0702 14:18:08.420823    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a3125bc_aec6_479f_9f15_be02e6406111.slice/crio-541410223b83c62ee80d22528a99f1fb0edfd5b934109aadf903f5243bd5460e WatchSource:0}: Error finding container 541410223b83c62ee80d22528a99f1fb0edfd5b934109aadf903f5243bd5460e: Status 404 returned error can't find the container with id 541410223b83c62ee80d22528a99f1fb0edfd5b934109aadf903f5243bd5460e
Jul 02 14:18:08.566452 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:08.566414    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws" event={"ID":"8a3125bc-aec6-479f-9f15-be02e6406111","Type":"ContainerStarted","Data":"d7459ea8b4df6631ba99194c2c9c3962f46622353a8c05d26e56e7cc7573800f"}
Jul 02 14:18:08.566933 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:08.566476    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws" event={"ID":"8a3125bc-aec6-479f-9f15-be02e6406111","Type":"ContainerStarted","Data":"541410223b83c62ee80d22528a99f1fb0edfd5b934109aadf903f5243bd5460e"}
Jul 02 14:18:08.568166 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:08.568135    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x" event={"ID":"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f","Type":"ContainerStarted","Data":"e9e0271bf66f50b666693a38296c2eb1f7921e8417f9151bd12ad9245242fdc7"}
Jul 02 14:18:08.568276 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:08.568193    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x" event={"ID":"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f","Type":"ContainerStarted","Data":"3cf4c440aa12741b58cc8f09ba27361692c3a99808f2c8fa4d9e6dffaa5eb2c4"}
Jul 02 14:18:09.575701 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:09.575605    2583 generic.go:356] "Generic (PLEG): container finished" podID="8a3125bc-aec6-479f-9f15-be02e6406111" containerID="d7459ea8b4df6631ba99194c2c9c3962f46622353a8c05d26e56e7cc7573800f" exitCode=0
Jul 02 14:18:09.576153 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:09.575692    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws" event={"ID":"8a3125bc-aec6-479f-9f15-be02e6406111","Type":"ContainerDied","Data":"d7459ea8b4df6631ba99194c2c9c3962f46622353a8c05d26e56e7cc7573800f"}
Jul 02 14:18:10.584229 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:10.584195    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws" event={"ID":"8a3125bc-aec6-479f-9f15-be02e6406111","Type":"ContainerStarted","Data":"6679ec39c94bdb416376ed650b64a50b399f8c1abbc5ac58de82218482da6220"}
Jul 02 14:18:10.584722 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:10.584314    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws"
Jul 02 14:18:10.606332 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:10.606279    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws" podStartSLOduration=2.893559356 podStartE2EDuration="3.606261516s" totalImagesPullingTime="0s" totalInitContainerRuntime="712.70216ms" isStatefulPod=false podCreationTimestamp="2026-07-02 14:18:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:18:10.605282963 +0000 UTC m=+3801.833194138" watchObservedRunningTime="2026-07-02 14:18:10.606261516 +0000 UTC m=+3801.834172690"
Jul 02 14:18:12.016479 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:12.016438    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:18:12.017348 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:12.017317    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c"]
Jul 02 14:18:12.119890 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:12.119849    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:18:12.596476 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:12.596370    2583 generic.go:356] "Generic (PLEG): container finished" podID="d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f" containerID="e9e0271bf66f50b666693a38296c2eb1f7921e8417f9151bd12ad9245242fdc7" exitCode=0
Jul 02 14:18:12.596476 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:12.596445    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x" event={"ID":"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f","Type":"ContainerDied","Data":"e9e0271bf66f50b666693a38296c2eb1f7921e8417f9151bd12ad9245242fdc7"}
Jul 02 14:18:13.603559 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:13.603522    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x" event={"ID":"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f","Type":"ContainerStarted","Data":"a5d5bcbf9ef03b48de1dd76bf4dfb6aa7c50cd077758e2d833289f2a27eac1b4"}
Jul 02 14:18:13.646083 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:13.646022    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x" podStartSLOduration=2.456445403 podStartE2EDuration="6.646002579s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.189557176s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:18:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:18:13.644030304 +0000 UTC m=+3804.871941481" watchObservedRunningTime="2026-07-02 14:18:13.646002579 +0000 UTC m=+3804.873913753"
Jul 02 14:18:13.836129 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:13.836091    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"
Jul 02 14:18:17.836403 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:17.836358    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"
Jul 02 14:18:17.838012 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:17.837982    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x" podUID="d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 02 14:18:27.039621 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:27.039571    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c" podUID="ba8f91e7-ae35-48af-b3ce-254f00abc9dd" containerName="main" containerID="cri-o://4ea9567789063454b12ec765c398d540f166dd0505981d2082be2be628c7080a" gracePeriod=45
Jul 02 14:18:27.133818 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:27.133769    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c" podUID="ba8f91e7-ae35-48af-b3ce-254f00abc9dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 02 14:18:27.302802 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:27.302728    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c"
Jul 02 14:18:27.324233 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:27.321820    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-tls-certs\") pod \"ba8f91e7-ae35-48af-b3ce-254f00abc9dd\" (UID: \"ba8f91e7-ae35-48af-b3ce-254f00abc9dd\") "
Jul 02 14:18:27.324233 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:27.321883    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-dshm\") pod \"ba8f91e7-ae35-48af-b3ce-254f00abc9dd\" (UID: \"ba8f91e7-ae35-48af-b3ce-254f00abc9dd\") "
Jul 02 14:18:27.324233 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:27.321932    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-tmp-dir\") pod \"ba8f91e7-ae35-48af-b3ce-254f00abc9dd\" (UID: \"ba8f91e7-ae35-48af-b3ce-254f00abc9dd\") "
Jul 02 14:18:27.324233 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:27.321971    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cr6kn\" (UniqueName: \"kubernetes.io/projected/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-kube-api-access-cr6kn\") pod \"ba8f91e7-ae35-48af-b3ce-254f00abc9dd\" (UID: \"ba8f91e7-ae35-48af-b3ce-254f00abc9dd\") "
Jul 02 14:18:27.324233 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:27.322529    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ba8f91e7-ae35-48af-b3ce-254f00abc9dd" (UID: "ba8f91e7-ae35-48af-b3ce-254f00abc9dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:18:27.326458 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:27.326414    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ba8f91e7-ae35-48af-b3ce-254f00abc9dd" (UID: "ba8f91e7-ae35-48af-b3ce-254f00abc9dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:18:27.326720 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:27.326656    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-dshm" (OuterVolumeSpecName: "dshm") pod "ba8f91e7-ae35-48af-b3ce-254f00abc9dd" (UID: "ba8f91e7-ae35-48af-b3ce-254f00abc9dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:18:27.326977 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:27.326952    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-kube-api-access-cr6kn" (OuterVolumeSpecName: "kube-api-access-cr6kn") pod "ba8f91e7-ae35-48af-b3ce-254f00abc9dd" (UID: "ba8f91e7-ae35-48af-b3ce-254f00abc9dd"). InnerVolumeSpecName "kube-api-access-cr6kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:18:27.422824 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:27.422792    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-home\") pod \"ba8f91e7-ae35-48af-b3ce-254f00abc9dd\" (UID: \"ba8f91e7-ae35-48af-b3ce-254f00abc9dd\") "
Jul 02 14:18:27.423011 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:27.422845    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-model-cache\") pod \"ba8f91e7-ae35-48af-b3ce-254f00abc9dd\" (UID: \"ba8f91e7-ae35-48af-b3ce-254f00abc9dd\") "
Jul 02 14:18:27.423081 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:27.423052    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-home" (OuterVolumeSpecName: "home") pod "ba8f91e7-ae35-48af-b3ce-254f00abc9dd" (UID: "ba8f91e7-ae35-48af-b3ce-254f00abc9dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:18:27.423137 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:27.423108    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-model-cache" (OuterVolumeSpecName: "model-cache") pod "ba8f91e7-ae35-48af-b3ce-254f00abc9dd" (UID: "ba8f91e7-ae35-48af-b3ce-254f00abc9dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:18:27.423246 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:27.423227    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-dshm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:18:27.423310 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:27.423250    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-tmp-dir\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:18:27.423310 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:27.423266    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr6kn\" (UniqueName: \"kubernetes.io/projected/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-kube-api-access-cr6kn\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:18:27.423310 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:27.423278    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-home\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:18:27.423310 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:27.423291    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:18:27.423310 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:27.423303    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba8f91e7-ae35-48af-b3ce-254f00abc9dd-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:18:27.679299 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:27.679265    2583 generic.go:356] "Generic (PLEG): container finished" podID="ba8f91e7-ae35-48af-b3ce-254f00abc9dd" containerID="4ea9567789063454b12ec765c398d540f166dd0505981d2082be2be628c7080a" exitCode=0
Jul 02 14:18:27.679522 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:27.679310    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c"
Jul 02 14:18:27.679522 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:27.679351    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c" event={"ID":"ba8f91e7-ae35-48af-b3ce-254f00abc9dd","Type":"ContainerDied","Data":"4ea9567789063454b12ec765c398d540f166dd0505981d2082be2be628c7080a"}
Jul 02 14:18:27.679522 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:27.679441    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c" event={"ID":"ba8f91e7-ae35-48af-b3ce-254f00abc9dd","Type":"ContainerDied","Data":"ab96e739924a2a0bd180001663da296da1b0eec0a783edbce1951c92a63abe8f"}
Jul 02 14:18:27.679522 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:27.679481    2583 scope.go:117] "RemoveContainer" containerID="4ea9567789063454b12ec765c398d540f166dd0505981d2082be2be628c7080a"
Jul 02 14:18:27.693724 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:27.693703    2583 scope.go:117] "RemoveContainer" containerID="4ea9567789063454b12ec765c398d540f166dd0505981d2082be2be628c7080a"
Jul 02 14:18:27.694046 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:18:27.694025    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ea9567789063454b12ec765c398d540f166dd0505981d2082be2be628c7080a\": container with ID starting with 4ea9567789063454b12ec765c398d540f166dd0505981d2082be2be628c7080a not found: ID does not exist" containerID="4ea9567789063454b12ec765c398d540f166dd0505981d2082be2be628c7080a"
Jul 02 14:18:27.694095 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:27.694058    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ea9567789063454b12ec765c398d540f166dd0505981d2082be2be628c7080a"} err="failed to get container status \"4ea9567789063454b12ec765c398d540f166dd0505981d2082be2be628c7080a\": rpc error: code = NotFound desc = could not find container \"4ea9567789063454b12ec765c398d540f166dd0505981d2082be2be628c7080a\": container with ID starting with 4ea9567789063454b12ec765c398d540f166dd0505981d2082be2be628c7080a not found: ID does not exist"
Jul 02 14:18:27.702420 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:27.702382    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c"]
Jul 02 14:18:27.706055 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:27.706027    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-nkx6c"]
Jul 02 14:18:27.836483 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:27.836439    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x" podUID="d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 02 14:18:28.049537 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:28.049439    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:18:28.108186 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:28.108134    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:18:29.234597 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:29.234562    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba8f91e7-ae35-48af-b3ce-254f00abc9dd" path="/var/lib/kubelet/pods/ba8f91e7-ae35-48af-b3ce-254f00abc9dd/volumes"
Jul 02 14:18:37.836361 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:37.836315    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x" podUID="d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 02 14:18:39.341346 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:39.341307    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:18:41.591337 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:41.591305    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws"
Jul 02 14:18:47.836018 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:47.835982    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x" podUID="d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 02 14:18:57.836605 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:18:57.836554    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x" podUID="d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 02 14:19:07.836209 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:07.836162    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x" podUID="d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 02 14:19:08.098507 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:08.098426    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9"]
Jul 02 14:19:08.098850 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:08.098836    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba8f91e7-ae35-48af-b3ce-254f00abc9dd" containerName="main"
Jul 02 14:19:08.098892 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:08.098852    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba8f91e7-ae35-48af-b3ce-254f00abc9dd" containerName="main"
Jul 02 14:19:08.098944 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:08.098934    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba8f91e7-ae35-48af-b3ce-254f00abc9dd" containerName="main"
Jul 02 14:19:08.101790 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:08.101764    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9"
Jul 02 14:19:08.104584 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:08.104561    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Jul 02 14:19:08.113999 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:08.113968    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9"]
Jul 02 14:19:08.122057 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:08.122027    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:19:08.132229 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:08.132189    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffa83d37-be0e-43be-a5a5-f45389503753-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9\" (UID: \"ffa83d37-be0e-43be-a5a5-f45389503753\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9"
Jul 02 14:19:08.132405 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:08.132242    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjtzp\" (UniqueName: \"kubernetes.io/projected/ffa83d37-be0e-43be-a5a5-f45389503753-kube-api-access-gjtzp\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9\" (UID: \"ffa83d37-be0e-43be-a5a5-f45389503753\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9"
Jul 02 14:19:08.132405 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:08.132337    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffa83d37-be0e-43be-a5a5-f45389503753-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9\" (UID: \"ffa83d37-be0e-43be-a5a5-f45389503753\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9"
Jul 02 14:19:08.132405 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:08.132366    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffa83d37-be0e-43be-a5a5-f45389503753-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9\" (UID: \"ffa83d37-be0e-43be-a5a5-f45389503753\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9"
Jul 02 14:19:08.132551 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:08.132436    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffa83d37-be0e-43be-a5a5-f45389503753-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9\" (UID: \"ffa83d37-be0e-43be-a5a5-f45389503753\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9"
Jul 02 14:19:08.132551 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:08.132519    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffa83d37-be0e-43be-a5a5-f45389503753-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9\" (UID: \"ffa83d37-be0e-43be-a5a5-f45389503753\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9"
Jul 02 14:19:08.233678 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:08.233634    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffa83d37-be0e-43be-a5a5-f45389503753-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9\" (UID: \"ffa83d37-be0e-43be-a5a5-f45389503753\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9"
Jul 02 14:19:08.233678 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:08.233681    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjtzp\" (UniqueName: \"kubernetes.io/projected/ffa83d37-be0e-43be-a5a5-f45389503753-kube-api-access-gjtzp\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9\" (UID: \"ffa83d37-be0e-43be-a5a5-f45389503753\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9"
Jul 02 14:19:08.233896 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:08.233734    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffa83d37-be0e-43be-a5a5-f45389503753-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9\" (UID: \"ffa83d37-be0e-43be-a5a5-f45389503753\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9"
Jul 02 14:19:08.233896 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:08.233752    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffa83d37-be0e-43be-a5a5-f45389503753-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9\" (UID: \"ffa83d37-be0e-43be-a5a5-f45389503753\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9"
Jul 02 14:19:08.233896 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:08.233784    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffa83d37-be0e-43be-a5a5-f45389503753-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9\" (UID: \"ffa83d37-be0e-43be-a5a5-f45389503753\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9"
Jul 02 14:19:08.233896 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:08.233853    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffa83d37-be0e-43be-a5a5-f45389503753-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9\" (UID: \"ffa83d37-be0e-43be-a5a5-f45389503753\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9"
Jul 02 14:19:08.234208 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:08.234177    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffa83d37-be0e-43be-a5a5-f45389503753-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9\" (UID: \"ffa83d37-be0e-43be-a5a5-f45389503753\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9"
Jul 02 14:19:08.234283 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:08.234204    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffa83d37-be0e-43be-a5a5-f45389503753-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9\" (UID: \"ffa83d37-be0e-43be-a5a5-f45389503753\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9"
Jul 02 14:19:08.234283 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:08.234224    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffa83d37-be0e-43be-a5a5-f45389503753-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9\" (UID: \"ffa83d37-be0e-43be-a5a5-f45389503753\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9"
Jul 02 14:19:08.236099 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:08.236070    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffa83d37-be0e-43be-a5a5-f45389503753-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9\" (UID: \"ffa83d37-be0e-43be-a5a5-f45389503753\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9"
Jul 02 14:19:08.236438 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:08.236417    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffa83d37-be0e-43be-a5a5-f45389503753-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9\" (UID: \"ffa83d37-be0e-43be-a5a5-f45389503753\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9"
Jul 02 14:19:08.244961 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:08.244933    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjtzp\" (UniqueName: \"kubernetes.io/projected/ffa83d37-be0e-43be-a5a5-f45389503753-kube-api-access-gjtzp\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9\" (UID: \"ffa83d37-be0e-43be-a5a5-f45389503753\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9"
Jul 02 14:19:08.412800 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:08.412759    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9"
Jul 02 14:19:08.433339 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:08.433284    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:19:08.563155 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:08.563124    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9"]
Jul 02 14:19:08.566209 ip-10-0-135-143 kubenswrapper[2583]: W0702 14:19:08.566179    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa83d37_be0e_43be_a5a5_f45389503753.slice/crio-9935c8c852c8c41effa648e914f5c91d6b214c4abcd12bb34f537ec3bbab7b73 WatchSource:0}: Error finding container 9935c8c852c8c41effa648e914f5c91d6b214c4abcd12bb34f537ec3bbab7b73: Status 404 returned error can't find the container with id 9935c8c852c8c41effa648e914f5c91d6b214c4abcd12bb34f537ec3bbab7b73
Jul 02 14:19:08.895470 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:08.895426    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9" event={"ID":"ffa83d37-be0e-43be-a5a5-f45389503753","Type":"ContainerStarted","Data":"74111a6279b320a66e6e9fb9fbc479d2672339e27cd7d090d079808d0487abe7"}
Jul 02 14:19:08.895987 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:08.895482    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9" event={"ID":"ffa83d37-be0e-43be-a5a5-f45389503753","Type":"ContainerStarted","Data":"9935c8c852c8c41effa648e914f5c91d6b214c4abcd12bb34f537ec3bbab7b73"}
Jul 02 14:19:08.916615 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:08.916565    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9" podStartSLOduration=0.916548578 podStartE2EDuration="916.548578ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:19:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:19:08.915554909 +0000 UTC m=+3860.143466083" watchObservedRunningTime="2026-07-02 14:19:08.916548578 +0000 UTC m=+3860.144459751"
Jul 02 14:19:09.413509 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:09.413459    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9"
Jul 02 14:19:17.836898 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:17.836808    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x" podUID="d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 02 14:19:18.413459 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:18.413417    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9"
Jul 02 14:19:18.425864 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:18.425828    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9"
Jul 02 14:19:18.961482 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:18.961452    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9"
Jul 02 14:19:27.836611 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:27.836565    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x" podUID="d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 02 14:19:32.726799 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:32.726757    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:19:37.836655 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:37.836606    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x" podUID="d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 02 14:19:41.036848 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:41.036798    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:19:47.836480 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:47.836427    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x" podUID="d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 02 14:19:49.599997 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:49.599963    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 14:19:49.610964 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:49.610938    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 14:19:57.836983 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:19:57.836934    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x" podUID="d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 02 14:20:07.836136 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:07.836084    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x" podUID="d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 02 14:20:17.846577 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:17.846545    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"
Jul 02 14:20:17.854535 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:17.854503    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"
Jul 02 14:20:23.191156 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:23.191117    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9"]
Jul 02 14:20:23.230732 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:23.230695    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:20:24.848302 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:24.848260    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:20:38.212885 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:38.212838    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9" podUID="ffa83d37-be0e-43be-a5a5-f45389503753" containerName="main" containerID="cri-o://74111a6279b320a66e6e9fb9fbc479d2672339e27cd7d090d079808d0487abe7" gracePeriod=45
Jul 02 14:20:38.369018 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:38.368986    2583 generic.go:356] "Generic (PLEG): container finished" podID="ffa83d37-be0e-43be-a5a5-f45389503753" containerID="74111a6279b320a66e6e9fb9fbc479d2672339e27cd7d090d079808d0487abe7" exitCode=0
Jul 02 14:20:38.369197 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:38.369042    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9" event={"ID":"ffa83d37-be0e-43be-a5a5-f45389503753","Type":"ContainerDied","Data":"74111a6279b320a66e6e9fb9fbc479d2672339e27cd7d090d079808d0487abe7"}
Jul 02 14:20:38.467607 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:38.467538    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9"
Jul 02 14:20:38.550943 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:38.550911    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffa83d37-be0e-43be-a5a5-f45389503753-model-cache\") pod \"ffa83d37-be0e-43be-a5a5-f45389503753\" (UID: \"ffa83d37-be0e-43be-a5a5-f45389503753\") "
Jul 02 14:20:38.551157 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:38.550971    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffa83d37-be0e-43be-a5a5-f45389503753-tls-certs\") pod \"ffa83d37-be0e-43be-a5a5-f45389503753\" (UID: \"ffa83d37-be0e-43be-a5a5-f45389503753\") "
Jul 02 14:20:38.551157 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:38.551018    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffa83d37-be0e-43be-a5a5-f45389503753-home\") pod \"ffa83d37-be0e-43be-a5a5-f45389503753\" (UID: \"ffa83d37-be0e-43be-a5a5-f45389503753\") "
Jul 02 14:20:38.551157 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:38.551044    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffa83d37-be0e-43be-a5a5-f45389503753-tmp-dir\") pod \"ffa83d37-be0e-43be-a5a5-f45389503753\" (UID: \"ffa83d37-be0e-43be-a5a5-f45389503753\") "
Jul 02 14:20:38.551157 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:38.551095    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffa83d37-be0e-43be-a5a5-f45389503753-dshm\") pod \"ffa83d37-be0e-43be-a5a5-f45389503753\" (UID: \"ffa83d37-be0e-43be-a5a5-f45389503753\") "
Jul 02 14:20:38.551377 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:38.551177    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjtzp\" (UniqueName: \"kubernetes.io/projected/ffa83d37-be0e-43be-a5a5-f45389503753-kube-api-access-gjtzp\") pod \"ffa83d37-be0e-43be-a5a5-f45389503753\" (UID: \"ffa83d37-be0e-43be-a5a5-f45389503753\") "
Jul 02 14:20:38.551377 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:38.551237    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffa83d37-be0e-43be-a5a5-f45389503753-model-cache" (OuterVolumeSpecName: "model-cache") pod "ffa83d37-be0e-43be-a5a5-f45389503753" (UID: "ffa83d37-be0e-43be-a5a5-f45389503753"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:20:38.551377 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:38.551290    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffa83d37-be0e-43be-a5a5-f45389503753-home" (OuterVolumeSpecName: "home") pod "ffa83d37-be0e-43be-a5a5-f45389503753" (UID: "ffa83d37-be0e-43be-a5a5-f45389503753"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:20:38.551377 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:38.551295    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffa83d37-be0e-43be-a5a5-f45389503753-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ffa83d37-be0e-43be-a5a5-f45389503753" (UID: "ffa83d37-be0e-43be-a5a5-f45389503753"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:20:38.551605 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:38.551572    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffa83d37-be0e-43be-a5a5-f45389503753-home\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:20:38.551658 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:38.551614    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffa83d37-be0e-43be-a5a5-f45389503753-tmp-dir\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:20:38.551658 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:38.551632    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffa83d37-be0e-43be-a5a5-f45389503753-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:20:38.553259 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:38.553227    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffa83d37-be0e-43be-a5a5-f45389503753-dshm" (OuterVolumeSpecName: "dshm") pod "ffa83d37-be0e-43be-a5a5-f45389503753" (UID: "ffa83d37-be0e-43be-a5a5-f45389503753"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:20:38.553728 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:38.553704    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffa83d37-be0e-43be-a5a5-f45389503753-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ffa83d37-be0e-43be-a5a5-f45389503753" (UID: "ffa83d37-be0e-43be-a5a5-f45389503753"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:20:38.553728 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:38.553716    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffa83d37-be0e-43be-a5a5-f45389503753-kube-api-access-gjtzp" (OuterVolumeSpecName: "kube-api-access-gjtzp") pod "ffa83d37-be0e-43be-a5a5-f45389503753" (UID: "ffa83d37-be0e-43be-a5a5-f45389503753"). InnerVolumeSpecName "kube-api-access-gjtzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:20:38.652222 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:38.652185    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffa83d37-be0e-43be-a5a5-f45389503753-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:20:38.652222 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:38.652215    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffa83d37-be0e-43be-a5a5-f45389503753-dshm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:20:38.652222 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:38.652225    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjtzp\" (UniqueName: \"kubernetes.io/projected/ffa83d37-be0e-43be-a5a5-f45389503753-kube-api-access-gjtzp\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:20:39.375657 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:39.375622    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9"
Jul 02 14:20:39.376079 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:39.375622    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9" event={"ID":"ffa83d37-be0e-43be-a5a5-f45389503753","Type":"ContainerDied","Data":"9935c8c852c8c41effa648e914f5c91d6b214c4abcd12bb34f537ec3bbab7b73"}
Jul 02 14:20:39.376079 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:39.375758    2583 scope.go:117] "RemoveContainer" containerID="74111a6279b320a66e6e9fb9fbc479d2672339e27cd7d090d079808d0487abe7"
Jul 02 14:20:39.393959 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:39.393923    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9"]
Jul 02 14:20:39.398145 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:39.398118    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4mmjt9"]
Jul 02 14:20:41.233707 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:41.233631    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffa83d37-be0e-43be-a5a5-f45389503753" path="/var/lib/kubelet/pods/ffa83d37-be0e-43be-a5a5-f45389503753/volumes"
Jul 02 14:20:49.843073 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:49.843040    2583 scope.go:117] "RemoveContainer" containerID="f57d850ec97fabc6fbfa4acc76bd2684375f0e6f34ccd0f98c48716f70d97cf3"
Jul 02 14:20:49.852260 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:20:49.852235    2583 scope.go:117] "RemoveContainer" containerID="eb6b368ec0a9363daab46b75a4e39ee07bfdfcd5d3d60f1ac1439e24487cf002"
Jul 02 14:21:17.881310 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:17.881273    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:21:17.882002 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:17.881979    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws"]
Jul 02 14:21:17.887200 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:17.887164    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"]
Jul 02 14:21:17.893354 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:21:17.893315    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 02 14:21:17.893354 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 14:21:17.893354 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 14:21:17.893354 ip-10-0-135-143 kubenswrapper[2583]:  > containerID="6679ec39c94bdb416376ed650b64a50b399f8c1abbc5ac58de82218482da6220" cmd=["/bin/sleep","15"]
Jul 02 14:21:17.893697 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:17.893373    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws" message="" err=<
Jul 02 14:21:17.893697 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 14:21:17.893697 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 14:21:17.893697 ip-10-0-135-143 kubenswrapper[2583]:  >
Jul 02 14:21:17.893697 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:21:17.893403    2583 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 02 14:21:17.893697 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 14:21:17.893697 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 14:21:17.893697 ip-10-0-135-143 kubenswrapper[2583]:  > pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws" podUID="8a3125bc-aec6-479f-9f15-be02e6406111" containerName="main" containerID="cri-o://6679ec39c94bdb416376ed650b64a50b399f8c1abbc5ac58de82218482da6220"
Jul 02 14:21:17.893697 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:17.893445    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws" podUID="8a3125bc-aec6-479f-9f15-be02e6406111" containerName="main" containerID="cri-o://6679ec39c94bdb416376ed650b64a50b399f8c1abbc5ac58de82218482da6220" gracePeriod=60
Jul 02 14:21:17.982730 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:17.982689    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:21:18.156058 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:18.156031    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws"
Jul 02 14:21:18.333430 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:18.333367    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4c5vn\" (UniqueName: \"kubernetes.io/projected/8a3125bc-aec6-479f-9f15-be02e6406111-kube-api-access-4c5vn\") pod \"8a3125bc-aec6-479f-9f15-be02e6406111\" (UID: \"8a3125bc-aec6-479f-9f15-be02e6406111\") "
Jul 02 14:21:18.333641 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:18.333454    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a3125bc-aec6-479f-9f15-be02e6406111-tls-certs\") pod \"8a3125bc-aec6-479f-9f15-be02e6406111\" (UID: \"8a3125bc-aec6-479f-9f15-be02e6406111\") "
Jul 02 14:21:18.333641 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:18.333484    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a3125bc-aec6-479f-9f15-be02e6406111-kserve-provision-location\") pod \"8a3125bc-aec6-479f-9f15-be02e6406111\" (UID: \"8a3125bc-aec6-479f-9f15-be02e6406111\") "
Jul 02 14:21:18.333641 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:18.333522    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8a3125bc-aec6-479f-9f15-be02e6406111-tokenizer-uds\") pod \"8a3125bc-aec6-479f-9f15-be02e6406111\" (UID: \"8a3125bc-aec6-479f-9f15-be02e6406111\") "
Jul 02 14:21:18.333872 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:18.333832    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3125bc-aec6-479f-9f15-be02e6406111-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "8a3125bc-aec6-479f-9f15-be02e6406111" (UID: "8a3125bc-aec6-479f-9f15-be02e6406111"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:21:18.334352 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:18.334317    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3125bc-aec6-479f-9f15-be02e6406111-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8a3125bc-aec6-479f-9f15-be02e6406111" (UID: "8a3125bc-aec6-479f-9f15-be02e6406111"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:21:18.335757 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:18.335734    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a3125bc-aec6-479f-9f15-be02e6406111-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a3125bc-aec6-479f-9f15-be02e6406111" (UID: "8a3125bc-aec6-479f-9f15-be02e6406111"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:21:18.335860 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:18.335838    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3125bc-aec6-479f-9f15-be02e6406111-kube-api-access-4c5vn" (OuterVolumeSpecName: "kube-api-access-4c5vn") pod "8a3125bc-aec6-479f-9f15-be02e6406111" (UID: "8a3125bc-aec6-479f-9f15-be02e6406111"). InnerVolumeSpecName "kube-api-access-4c5vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:21:18.434571 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:18.434462    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a3125bc-aec6-479f-9f15-be02e6406111-kserve-provision-location\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:21:18.434571 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:18.434508    2583 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8a3125bc-aec6-479f-9f15-be02e6406111-tokenizer-uds\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:21:18.434571 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:18.434522    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c5vn\" (UniqueName: \"kubernetes.io/projected/8a3125bc-aec6-479f-9f15-be02e6406111-kube-api-access-4c5vn\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:21:18.434571 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:18.434535    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a3125bc-aec6-479f-9f15-be02e6406111-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:21:18.574246 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:18.574205    2583 generic.go:356] "Generic (PLEG): container finished" podID="8a3125bc-aec6-479f-9f15-be02e6406111" containerID="6679ec39c94bdb416376ed650b64a50b399f8c1abbc5ac58de82218482da6220" exitCode=0
Jul 02 14:21:18.574445 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:18.574261    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws"
Jul 02 14:21:18.574445 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:18.574287    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws" event={"ID":"8a3125bc-aec6-479f-9f15-be02e6406111","Type":"ContainerDied","Data":"6679ec39c94bdb416376ed650b64a50b399f8c1abbc5ac58de82218482da6220"}
Jul 02 14:21:18.574445 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:18.574339    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws" event={"ID":"8a3125bc-aec6-479f-9f15-be02e6406111","Type":"ContainerDied","Data":"541410223b83c62ee80d22528a99f1fb0edfd5b934109aadf903f5243bd5460e"}
Jul 02 14:21:18.574445 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:18.574358    2583 scope.go:117] "RemoveContainer" containerID="6679ec39c94bdb416376ed650b64a50b399f8c1abbc5ac58de82218482da6220"
Jul 02 14:21:18.583976 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:18.583950    2583 scope.go:117] "RemoveContainer" containerID="d7459ea8b4df6631ba99194c2c9c3962f46622353a8c05d26e56e7cc7573800f"
Jul 02 14:21:18.592592 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:18.592571    2583 scope.go:117] "RemoveContainer" containerID="6679ec39c94bdb416376ed650b64a50b399f8c1abbc5ac58de82218482da6220"
Jul 02 14:21:18.592872 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:21:18.592853    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6679ec39c94bdb416376ed650b64a50b399f8c1abbc5ac58de82218482da6220\": container with ID starting with 6679ec39c94bdb416376ed650b64a50b399f8c1abbc5ac58de82218482da6220 not found: ID does not exist" containerID="6679ec39c94bdb416376ed650b64a50b399f8c1abbc5ac58de82218482da6220"
Jul 02 14:21:18.592953 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:18.592883    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6679ec39c94bdb416376ed650b64a50b399f8c1abbc5ac58de82218482da6220"} err="failed to get container status \"6679ec39c94bdb416376ed650b64a50b399f8c1abbc5ac58de82218482da6220\": rpc error: code = NotFound desc = could not find container \"6679ec39c94bdb416376ed650b64a50b399f8c1abbc5ac58de82218482da6220\": container with ID starting with 6679ec39c94bdb416376ed650b64a50b399f8c1abbc5ac58de82218482da6220 not found: ID does not exist"
Jul 02 14:21:18.592953 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:18.592909    2583 scope.go:117] "RemoveContainer" containerID="d7459ea8b4df6631ba99194c2c9c3962f46622353a8c05d26e56e7cc7573800f"
Jul 02 14:21:18.593155 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:21:18.593137    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7459ea8b4df6631ba99194c2c9c3962f46622353a8c05d26e56e7cc7573800f\": container with ID starting with d7459ea8b4df6631ba99194c2c9c3962f46622353a8c05d26e56e7cc7573800f not found: ID does not exist" containerID="d7459ea8b4df6631ba99194c2c9c3962f46622353a8c05d26e56e7cc7573800f"
Jul 02 14:21:18.593195 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:18.593161    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7459ea8b4df6631ba99194c2c9c3962f46622353a8c05d26e56e7cc7573800f"} err="failed to get container status \"d7459ea8b4df6631ba99194c2c9c3962f46622353a8c05d26e56e7cc7573800f\": rpc error: code = NotFound desc = could not find container \"d7459ea8b4df6631ba99194c2c9c3962f46622353a8c05d26e56e7cc7573800f\": container with ID starting with d7459ea8b4df6631ba99194c2c9c3962f46622353a8c05d26e56e7cc7573800f not found: ID does not exist"
Jul 02 14:21:18.598165 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:18.598140    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws"]
Jul 02 14:21:18.604495 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:18.604473    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c6cfc4c67-fw5ws"]
Jul 02 14:21:19.234028 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:19.233984    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a3125bc-aec6-479f-9f15-be02e6406111" path="/var/lib/kubelet/pods/8a3125bc-aec6-479f-9f15-be02e6406111/volumes"
Jul 02 14:21:24.607192 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.607162    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6"]
Jul 02 14:21:24.607721 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.607592    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a3125bc-aec6-479f-9f15-be02e6406111" containerName="main"
Jul 02 14:21:24.607721 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.607607    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a3125bc-aec6-479f-9f15-be02e6406111" containerName="main"
Jul 02 14:21:24.607721 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.607616    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffa83d37-be0e-43be-a5a5-f45389503753" containerName="main"
Jul 02 14:21:24.607721 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.607622    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffa83d37-be0e-43be-a5a5-f45389503753" containerName="main"
Jul 02 14:21:24.607721 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.607653    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a3125bc-aec6-479f-9f15-be02e6406111" containerName="storage-initializer"
Jul 02 14:21:24.607721 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.607660    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a3125bc-aec6-479f-9f15-be02e6406111" containerName="storage-initializer"
Jul 02 14:21:24.607721 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.607714    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a3125bc-aec6-479f-9f15-be02e6406111" containerName="main"
Jul 02 14:21:24.607953 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.607726    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffa83d37-be0e-43be-a5a5-f45389503753" containerName="main"
Jul 02 14:21:24.612162 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.612142    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6"
Jul 02 14:21:24.615575 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.615554    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 02 14:21:24.625873 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.625848    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6"]
Jul 02 14:21:24.673690 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.673637    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:21:24.797374 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.797334    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6\" (UID: \"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6"
Jul 02 14:21:24.797596 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.797382    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-model-cache\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6\" (UID: \"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6"
Jul 02 14:21:24.797596 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.797447    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvbsq\" (UniqueName: \"kubernetes.io/projected/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-kube-api-access-nvbsq\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6\" (UID: \"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6"
Jul 02 14:21:24.797596 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.797473    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6\" (UID: \"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6"
Jul 02 14:21:24.797596 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.797500    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-dshm\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6\" (UID: \"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6"
Jul 02 14:21:24.797596 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.797564    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-home\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6\" (UID: \"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6"
Jul 02 14:21:24.889679 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.889638    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2"]
Jul 02 14:21:24.893175 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.893150    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2"
Jul 02 14:21:24.896269 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.896248    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-dkc4f\""
Jul 02 14:21:24.898953 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.898930    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6\" (UID: \"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6"
Jul 02 14:21:24.899073 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.898977    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-model-cache\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6\" (UID: \"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6"
Jul 02 14:21:24.899073 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.899015    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvbsq\" (UniqueName: \"kubernetes.io/projected/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-kube-api-access-nvbsq\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6\" (UID: \"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6"
Jul 02 14:21:24.899073 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.899037    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6\" (UID: \"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6"
Jul 02 14:21:24.899073 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.899056    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a160b04-7033-4df7-b78e-1f7759014ae3-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2\" (UID: \"0a160b04-7033-4df7-b78e-1f7759014ae3\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2"
Jul 02 14:21:24.899295 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.899120    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c4zk\" (UniqueName: \"kubernetes.io/projected/0a160b04-7033-4df7-b78e-1f7759014ae3-kube-api-access-9c4zk\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2\" (UID: \"0a160b04-7033-4df7-b78e-1f7759014ae3\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2"
Jul 02 14:21:24.899295 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.899185    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-dshm\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6\" (UID: \"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6"
Jul 02 14:21:24.899295 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.899253    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0a160b04-7033-4df7-b78e-1f7759014ae3-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2\" (UID: \"0a160b04-7033-4df7-b78e-1f7759014ae3\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2"
Jul 02 14:21:24.899295 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.899282    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-home\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6\" (UID: \"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6"
Jul 02 14:21:24.899556 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.899416    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6\" (UID: \"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6"
Jul 02 14:21:24.899616 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.899569    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-model-cache\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6\" (UID: \"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6"
Jul 02 14:21:24.899673 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.899645    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-home\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6\" (UID: \"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6"
Jul 02 14:21:24.901910 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.901856    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-dshm\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6\" (UID: \"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6"
Jul 02 14:21:24.902298 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.902272    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6\" (UID: \"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6"
Jul 02 14:21:24.905239 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.905218    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2"]
Jul 02 14:21:24.908467 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.908444    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvbsq\" (UniqueName: \"kubernetes.io/projected/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-kube-api-access-nvbsq\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6\" (UID: \"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6"
Jul 02 14:21:24.922271 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.922236    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6"
Jul 02 14:21:24.999941 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.999901    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c4zk\" (UniqueName: \"kubernetes.io/projected/0a160b04-7033-4df7-b78e-1f7759014ae3-kube-api-access-9c4zk\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2\" (UID: \"0a160b04-7033-4df7-b78e-1f7759014ae3\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2"
Jul 02 14:21:25.000128 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:24.999953    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0a160b04-7033-4df7-b78e-1f7759014ae3-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2\" (UID: \"0a160b04-7033-4df7-b78e-1f7759014ae3\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2"
Jul 02 14:21:25.000128 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:25.000019    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a160b04-7033-4df7-b78e-1f7759014ae3-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2\" (UID: \"0a160b04-7033-4df7-b78e-1f7759014ae3\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2"
Jul 02 14:21:25.000560 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:25.000532    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0a160b04-7033-4df7-b78e-1f7759014ae3-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2\" (UID: \"0a160b04-7033-4df7-b78e-1f7759014ae3\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2"
Jul 02 14:21:25.002369 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:25.002350    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a160b04-7033-4df7-b78e-1f7759014ae3-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2\" (UID: \"0a160b04-7033-4df7-b78e-1f7759014ae3\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2"
Jul 02 14:21:25.010410 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:25.010365    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c4zk\" (UniqueName: \"kubernetes.io/projected/0a160b04-7033-4df7-b78e-1f7759014ae3-kube-api-access-9c4zk\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2\" (UID: \"0a160b04-7033-4df7-b78e-1f7759014ae3\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2"
Jul 02 14:21:25.068400 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:25.068360    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6"]
Jul 02 14:21:25.070435 ip-10-0-135-143 kubenswrapper[2583]: W0702 14:21:25.070408    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86f61e5c_51e7_4a88_b1fb_9df7aa1bc227.slice/crio-df30929e3a8a9c30a057a0ee5f6b7ef664ce62ea2af41945b020340ffbef58ce WatchSource:0}: Error finding container df30929e3a8a9c30a057a0ee5f6b7ef664ce62ea2af41945b020340ffbef58ce: Status 404 returned error can't find the container with id df30929e3a8a9c30a057a0ee5f6b7ef664ce62ea2af41945b020340ffbef58ce
Jul 02 14:21:25.142806 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:25.142713    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:21:25.204198 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:25.204166    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2"
Jul 02 14:21:25.349268 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:25.349238    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2"]
Jul 02 14:21:25.350775 ip-10-0-135-143 kubenswrapper[2583]: W0702 14:21:25.350746    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a160b04_7033_4df7_b78e_1f7759014ae3.slice/crio-2013a7b4772cc97d9e848f67bc72436fdeb052f6d1a100fb2284a1de75f76949 WatchSource:0}: Error finding container 2013a7b4772cc97d9e848f67bc72436fdeb052f6d1a100fb2284a1de75f76949: Status 404 returned error can't find the container with id 2013a7b4772cc97d9e848f67bc72436fdeb052f6d1a100fb2284a1de75f76949
Jul 02 14:21:25.616629 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:25.616588    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2" event={"ID":"0a160b04-7033-4df7-b78e-1f7759014ae3","Type":"ContainerStarted","Data":"ceb7069f1878573f0204a37d301ecd64977b0e80d2ec38b76e33da9f85a0b0c1"}
Jul 02 14:21:25.617120 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:25.616646    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2" event={"ID":"0a160b04-7033-4df7-b78e-1f7759014ae3","Type":"ContainerStarted","Data":"2013a7b4772cc97d9e848f67bc72436fdeb052f6d1a100fb2284a1de75f76949"}
Jul 02 14:21:25.617120 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:25.616682    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2"
Jul 02 14:21:25.618367 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:25.618342    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6" event={"ID":"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227","Type":"ContainerStarted","Data":"3a532f85a80d18e026d95c92d13484fca2e7c31b00739ab0c55899c0b4d72090"}
Jul 02 14:21:25.618489 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:25.618416    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6" event={"ID":"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227","Type":"ContainerStarted","Data":"df30929e3a8a9c30a057a0ee5f6b7ef664ce62ea2af41945b020340ffbef58ce"}
Jul 02 14:21:25.636252 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:25.636200    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2" podStartSLOduration=1.636183866 podStartE2EDuration="1.636183866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:21:25.635086637 +0000 UTC m=+3996.862997813" watchObservedRunningTime="2026-07-02 14:21:25.636183866 +0000 UTC m=+3996.864095039"
Jul 02 14:21:25.661314 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:25.661219    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6" podStartSLOduration=1.661201967 podStartE2EDuration="1.661201967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:21:25.660046387 +0000 UTC m=+3996.887957561" watchObservedRunningTime="2026-07-02 14:21:25.661201967 +0000 UTC m=+3996.889113140"
Jul 02 14:21:25.922634 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:25.922545    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6"
Jul 02 14:21:32.912767 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:32.912706    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x" podUID="d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f" containerName="main" containerID="cri-o://a5d5bcbf9ef03b48de1dd76bf4dfb6aa7c50cd077758e2d833289f2a27eac1b4" gracePeriod=45
Jul 02 14:21:33.664259 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:33.664222    2583 generic.go:356] "Generic (PLEG): container finished" podID="d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f" containerID="a5d5bcbf9ef03b48de1dd76bf4dfb6aa7c50cd077758e2d833289f2a27eac1b4" exitCode=0
Jul 02 14:21:33.664430 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:33.664300    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x" event={"ID":"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f","Type":"ContainerDied","Data":"a5d5bcbf9ef03b48de1dd76bf4dfb6aa7c50cd077758e2d833289f2a27eac1b4"}
Jul 02 14:21:33.664430 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:33.664364    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x" event={"ID":"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f","Type":"ContainerDied","Data":"3cf4c440aa12741b58cc8f09ba27361692c3a99808f2c8fa4d9e6dffaa5eb2c4"}
Jul 02 14:21:33.664430 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:33.664405    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cf4c440aa12741b58cc8f09ba27361692c3a99808f2c8fa4d9e6dffaa5eb2c4"
Jul 02 14:21:33.673452 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:33.673428    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"
Jul 02 14:21:33.777062 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:33.776960    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5gr8\" (UniqueName: \"kubernetes.io/projected/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-kube-api-access-q5gr8\") pod \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\" (UID: \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\") "
Jul 02 14:21:33.777062 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:33.777028    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-home\") pod \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\" (UID: \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\") "
Jul 02 14:21:33.777062 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:33.777057    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-model-cache\") pod \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\" (UID: \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\") "
Jul 02 14:21:33.777354 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:33.777091    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-tls-certs\") pod \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\" (UID: \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\") "
Jul 02 14:21:33.777354 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:33.777181    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-tmp-dir\") pod \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\" (UID: \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\") "
Jul 02 14:21:33.777354 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:33.777207    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-dshm\") pod \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\" (UID: \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\") "
Jul 02 14:21:33.777354 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:33.777233    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-kserve-provision-location\") pod \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\" (UID: \"d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f\") "
Jul 02 14:21:33.777354 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:33.777330    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-model-cache" (OuterVolumeSpecName: "model-cache") pod "d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f" (UID: "d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:21:33.777698 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:33.777676    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:21:33.777885 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:33.777841    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-home" (OuterVolumeSpecName: "home") pod "d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f" (UID: "d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:21:33.779282 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:33.779257    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-kube-api-access-q5gr8" (OuterVolumeSpecName: "kube-api-access-q5gr8") pod "d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f" (UID: "d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f"). InnerVolumeSpecName "kube-api-access-q5gr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:21:33.779455 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:33.779434    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-dshm" (OuterVolumeSpecName: "dshm") pod "d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f" (UID: "d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:21:33.779719 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:33.779694    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f" (UID: "d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:21:33.788821 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:33.788766    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f" (UID: "d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:21:33.841180 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:33.841132    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f" (UID: "d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:21:33.879150 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:33.879110    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-tmp-dir\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:21:33.879150 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:33.879144    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-dshm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:21:33.879150 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:33.879159    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-kserve-provision-location\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:21:33.879424 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:33.879172    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5gr8\" (UniqueName: \"kubernetes.io/projected/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-kube-api-access-q5gr8\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:21:33.879424 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:33.879183    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-home\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:21:33.879424 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:33.879194    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:21:34.670376 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:34.670337    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"
Jul 02 14:21:34.694949 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:34.694913    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"]
Jul 02 14:21:34.698075 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:34.698049    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-757c67794-wrk7x"]
Jul 02 14:21:34.922956 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:34.922866    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6"
Jul 02 14:21:34.935602 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:34.935573    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6"
Jul 02 14:21:35.233992 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:35.233911    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f" path="/var/lib/kubelet/pods/d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f/volumes"
Jul 02 14:21:35.693686 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:35.693656    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6"
Jul 02 14:21:52.869827 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:52.869785    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:21:56.625983 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:21:56.625949    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2"
Jul 02 14:22:07.660862 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.660829    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"]
Jul 02 14:22:07.661245 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.661229    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f" containerName="storage-initializer"
Jul 02 14:22:07.661291 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.661246    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f" containerName="storage-initializer"
Jul 02 14:22:07.661291 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.661265    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f" containerName="main"
Jul 02 14:22:07.661291 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.661272    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f" containerName="main"
Jul 02 14:22:07.661427 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.661342    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0fac6a5-c73f-4855-a0a6-c8a4b0a3312f" containerName="main"
Jul 02 14:22:07.664635 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.664615    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"
Jul 02 14:22:07.669089 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.669070    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Jul 02 14:22:07.669208 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.669188    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 14:22:07.669339 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.669323    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 14:22:07.689738 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.689709    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"]
Jul 02 14:22:07.806694 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.806653    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22d8d47-875a-4a2b-964a-034eacb7ee77-home\") pod \"lora-single-adapter-test-kserve-6cfb96b7d7-g29ld\" (UID: \"a22d8d47-875a-4a2b-964a-034eacb7ee77\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"
Jul 02 14:22:07.806888 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.806726    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22d8d47-875a-4a2b-964a-034eacb7ee77-tmp-dir\") pod \"lora-single-adapter-test-kserve-6cfb96b7d7-g29ld\" (UID: \"a22d8d47-875a-4a2b-964a-034eacb7ee77\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"
Jul 02 14:22:07.806888 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.806789    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr5lk\" (UniqueName: \"kubernetes.io/projected/a22d8d47-875a-4a2b-964a-034eacb7ee77-kube-api-access-gr5lk\") pod \"lora-single-adapter-test-kserve-6cfb96b7d7-g29ld\" (UID: \"a22d8d47-875a-4a2b-964a-034eacb7ee77\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"
Jul 02 14:22:07.806888 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.806821    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a22d8d47-875a-4a2b-964a-034eacb7ee77-cabundle-cert\") pod \"lora-single-adapter-test-kserve-6cfb96b7d7-g29ld\" (UID: \"a22d8d47-875a-4a2b-964a-034eacb7ee77\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"
Jul 02 14:22:07.806888 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.806844    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22d8d47-875a-4a2b-964a-034eacb7ee77-tls-certs\") pod \"lora-single-adapter-test-kserve-6cfb96b7d7-g29ld\" (UID: \"a22d8d47-875a-4a2b-964a-034eacb7ee77\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"
Jul 02 14:22:07.807041 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.806908    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22d8d47-875a-4a2b-964a-034eacb7ee77-model-cache\") pod \"lora-single-adapter-test-kserve-6cfb96b7d7-g29ld\" (UID: \"a22d8d47-875a-4a2b-964a-034eacb7ee77\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"
Jul 02 14:22:07.807041 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.806943    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a22d8d47-875a-4a2b-964a-034eacb7ee77-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-6cfb96b7d7-g29ld\" (UID: \"a22d8d47-875a-4a2b-964a-034eacb7ee77\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"
Jul 02 14:22:07.807041 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.806967    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22d8d47-875a-4a2b-964a-034eacb7ee77-dshm\") pod \"lora-single-adapter-test-kserve-6cfb96b7d7-g29ld\" (UID: \"a22d8d47-875a-4a2b-964a-034eacb7ee77\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"
Jul 02 14:22:07.907501 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.907461    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22d8d47-875a-4a2b-964a-034eacb7ee77-model-cache\") pod \"lora-single-adapter-test-kserve-6cfb96b7d7-g29ld\" (UID: \"a22d8d47-875a-4a2b-964a-034eacb7ee77\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"
Jul 02 14:22:07.907501 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.907503    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a22d8d47-875a-4a2b-964a-034eacb7ee77-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-6cfb96b7d7-g29ld\" (UID: \"a22d8d47-875a-4a2b-964a-034eacb7ee77\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"
Jul 02 14:22:07.907767 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.907523    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22d8d47-875a-4a2b-964a-034eacb7ee77-dshm\") pod \"lora-single-adapter-test-kserve-6cfb96b7d7-g29ld\" (UID: \"a22d8d47-875a-4a2b-964a-034eacb7ee77\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"
Jul 02 14:22:07.907767 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.907562    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22d8d47-875a-4a2b-964a-034eacb7ee77-home\") pod \"lora-single-adapter-test-kserve-6cfb96b7d7-g29ld\" (UID: \"a22d8d47-875a-4a2b-964a-034eacb7ee77\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"
Jul 02 14:22:07.907767 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.907607    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22d8d47-875a-4a2b-964a-034eacb7ee77-tmp-dir\") pod \"lora-single-adapter-test-kserve-6cfb96b7d7-g29ld\" (UID: \"a22d8d47-875a-4a2b-964a-034eacb7ee77\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"
Jul 02 14:22:07.907767 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.907675    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr5lk\" (UniqueName: \"kubernetes.io/projected/a22d8d47-875a-4a2b-964a-034eacb7ee77-kube-api-access-gr5lk\") pod \"lora-single-adapter-test-kserve-6cfb96b7d7-g29ld\" (UID: \"a22d8d47-875a-4a2b-964a-034eacb7ee77\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"
Jul 02 14:22:07.907767 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.907709    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a22d8d47-875a-4a2b-964a-034eacb7ee77-cabundle-cert\") pod \"lora-single-adapter-test-kserve-6cfb96b7d7-g29ld\" (UID: \"a22d8d47-875a-4a2b-964a-034eacb7ee77\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"
Jul 02 14:22:07.907767 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.907737    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22d8d47-875a-4a2b-964a-034eacb7ee77-tls-certs\") pod \"lora-single-adapter-test-kserve-6cfb96b7d7-g29ld\" (UID: \"a22d8d47-875a-4a2b-964a-034eacb7ee77\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"
Jul 02 14:22:07.908049 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.907886    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22d8d47-875a-4a2b-964a-034eacb7ee77-model-cache\") pod \"lora-single-adapter-test-kserve-6cfb96b7d7-g29ld\" (UID: \"a22d8d47-875a-4a2b-964a-034eacb7ee77\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"
Jul 02 14:22:07.908049 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.907997    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a22d8d47-875a-4a2b-964a-034eacb7ee77-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-6cfb96b7d7-g29ld\" (UID: \"a22d8d47-875a-4a2b-964a-034eacb7ee77\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"
Jul 02 14:22:07.908139 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.908095    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22d8d47-875a-4a2b-964a-034eacb7ee77-home\") pod \"lora-single-adapter-test-kserve-6cfb96b7d7-g29ld\" (UID: \"a22d8d47-875a-4a2b-964a-034eacb7ee77\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"
Jul 02 14:22:07.908272 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.908253    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22d8d47-875a-4a2b-964a-034eacb7ee77-tmp-dir\") pod \"lora-single-adapter-test-kserve-6cfb96b7d7-g29ld\" (UID: \"a22d8d47-875a-4a2b-964a-034eacb7ee77\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"
Jul 02 14:22:07.908559 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.908534    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a22d8d47-875a-4a2b-964a-034eacb7ee77-cabundle-cert\") pod \"lora-single-adapter-test-kserve-6cfb96b7d7-g29ld\" (UID: \"a22d8d47-875a-4a2b-964a-034eacb7ee77\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"
Jul 02 14:22:07.910038 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.910013    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22d8d47-875a-4a2b-964a-034eacb7ee77-dshm\") pod \"lora-single-adapter-test-kserve-6cfb96b7d7-g29ld\" (UID: \"a22d8d47-875a-4a2b-964a-034eacb7ee77\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"
Jul 02 14:22:07.910214 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.910196    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22d8d47-875a-4a2b-964a-034eacb7ee77-tls-certs\") pod \"lora-single-adapter-test-kserve-6cfb96b7d7-g29ld\" (UID: \"a22d8d47-875a-4a2b-964a-034eacb7ee77\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"
Jul 02 14:22:07.917357 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.917284    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr5lk\" (UniqueName: \"kubernetes.io/projected/a22d8d47-875a-4a2b-964a-034eacb7ee77-kube-api-access-gr5lk\") pod \"lora-single-adapter-test-kserve-6cfb96b7d7-g29ld\" (UID: \"a22d8d47-875a-4a2b-964a-034eacb7ee77\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"
Jul 02 14:22:07.974973 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:07.974925    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"
Jul 02 14:22:08.135243 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:08.135123    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"]
Jul 02 14:22:08.138430 ip-10-0-135-143 kubenswrapper[2583]: W0702 14:22:08.138375    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda22d8d47_875a_4a2b_964a_034eacb7ee77.slice/crio-5a90934a6f97a83bfd81d11bd2af18ebb654dccce79332b01bca85382ca55dca WatchSource:0}: Error finding container 5a90934a6f97a83bfd81d11bd2af18ebb654dccce79332b01bca85382ca55dca: Status 404 returned error can't find the container with id 5a90934a6f97a83bfd81d11bd2af18ebb654dccce79332b01bca85382ca55dca
Jul 02 14:22:08.854105 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:08.854069    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld" event={"ID":"a22d8d47-875a-4a2b-964a-034eacb7ee77","Type":"ContainerStarted","Data":"f4b480bd2f479aa033ea405a82a509f828f23ef3ec77dd8149d007a2911870d5"}
Jul 02 14:22:08.854565 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:08.854131    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld" event={"ID":"a22d8d47-875a-4a2b-964a-034eacb7ee77","Type":"ContainerStarted","Data":"5a90934a6f97a83bfd81d11bd2af18ebb654dccce79332b01bca85382ca55dca"}
Jul 02 14:22:08.972657 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:08.972617    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:22:12.885850 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:12.885817    2583 generic.go:356] "Generic (PLEG): container finished" podID="a22d8d47-875a-4a2b-964a-034eacb7ee77" containerID="f4b480bd2f479aa033ea405a82a509f828f23ef3ec77dd8149d007a2911870d5" exitCode=0
Jul 02 14:22:12.886274 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:12.885880    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld" event={"ID":"a22d8d47-875a-4a2b-964a-034eacb7ee77","Type":"ContainerDied","Data":"f4b480bd2f479aa033ea405a82a509f828f23ef3ec77dd8149d007a2911870d5"}
Jul 02 14:22:13.901256 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:13.901217    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld" event={"ID":"a22d8d47-875a-4a2b-964a-034eacb7ee77","Type":"ContainerStarted","Data":"336cd154becf8ead8d1b26ea9c517e1edfb5fbde6b86b8e446fc4b3e7b21d8e4"}
Jul 02 14:22:13.928968 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:13.928911    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld" podStartSLOduration=2.634434112 podStartE2EDuration="6.92888828s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.294454168s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:22:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:22:13.927798777 +0000 UTC m=+4045.155709952" watchObservedRunningTime="2026-07-02 14:22:13.92888828 +0000 UTC m=+4045.156799455"
Jul 02 14:22:13.975593 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:13.975553    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"
Jul 02 14:22:17.976049 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:17.976009    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"
Jul 02 14:22:17.977666 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:17.977634    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld" podUID="a22d8d47-875a-4a2b-964a-034eacb7ee77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 02 14:22:27.975838 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:27.975796    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld" podUID="a22d8d47-875a-4a2b-964a-034eacb7ee77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 02 14:22:37.975715 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:37.975670    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld" podUID="a22d8d47-875a-4a2b-964a-034eacb7ee77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 02 14:22:43.196739 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:43.196703    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:22:43.297091 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:43.297047    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:22:47.976088 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:47.976041    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld" podUID="a22d8d47-875a-4a2b-964a-034eacb7ee77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 02 14:22:57.976085 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:22:57.976037    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld" podUID="a22d8d47-875a-4a2b-964a-034eacb7ee77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 02 14:23:07.975664 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:07.975614    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld" podUID="a22d8d47-875a-4a2b-964a-034eacb7ee77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 02 14:23:12.726084 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:12.726045    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2"]
Jul 02 14:23:12.727118 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:12.727081    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:23:12.733315 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:12.733286    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6"]
Jul 02 14:23:12.738500 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:23:12.738452    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 02 14:23:12.738500 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 14:23:12.738500 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 14:23:12.738500 ip-10-0-135-143 kubenswrapper[2583]:  > containerID="ceb7069f1878573f0204a37d301ecd64977b0e80d2ec38b76e33da9f85a0b0c1" cmd=["/bin/sleep","15"]
Jul 02 14:23:12.738817 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:12.738517    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2" message="" err=<
Jul 02 14:23:12.738817 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 14:23:12.738817 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 14:23:12.738817 ip-10-0-135-143 kubenswrapper[2583]:  >
Jul 02 14:23:12.738817 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:23:12.738536    2583 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 02 14:23:12.738817 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 14:23:12.738817 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 14:23:12.738817 ip-10-0-135-143 kubenswrapper[2583]:  > pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2" podUID="0a160b04-7033-4df7-b78e-1f7759014ae3" containerName="main" containerID="cri-o://ceb7069f1878573f0204a37d301ecd64977b0e80d2ec38b76e33da9f85a0b0c1"
Jul 02 14:23:12.738817 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:12.738617    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2" podUID="0a160b04-7033-4df7-b78e-1f7759014ae3" containerName="main" containerID="cri-o://ceb7069f1878573f0204a37d301ecd64977b0e80d2ec38b76e33da9f85a0b0c1" gracePeriod=60
Jul 02 14:23:12.854075 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:12.854039    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:23:13.016621 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:13.016591    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2"
Jul 02 14:23:13.168754 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:13.168714    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9c4zk\" (UniqueName: \"kubernetes.io/projected/0a160b04-7033-4df7-b78e-1f7759014ae3-kube-api-access-9c4zk\") pod \"0a160b04-7033-4df7-b78e-1f7759014ae3\" (UID: \"0a160b04-7033-4df7-b78e-1f7759014ae3\") "
Jul 02 14:23:13.168934 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:13.168781    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0a160b04-7033-4df7-b78e-1f7759014ae3-tokenizer-uds\") pod \"0a160b04-7033-4df7-b78e-1f7759014ae3\" (UID: \"0a160b04-7033-4df7-b78e-1f7759014ae3\") "
Jul 02 14:23:13.168934 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:13.168912    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a160b04-7033-4df7-b78e-1f7759014ae3-tls-certs\") pod \"0a160b04-7033-4df7-b78e-1f7759014ae3\" (UID: \"0a160b04-7033-4df7-b78e-1f7759014ae3\") "
Jul 02 14:23:13.169092 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:13.169048    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a160b04-7033-4df7-b78e-1f7759014ae3-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "0a160b04-7033-4df7-b78e-1f7759014ae3" (UID: "0a160b04-7033-4df7-b78e-1f7759014ae3"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:23:13.169234 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:13.169193    2583 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0a160b04-7033-4df7-b78e-1f7759014ae3-tokenizer-uds\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:23:13.170880 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:13.170849    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a160b04-7033-4df7-b78e-1f7759014ae3-kube-api-access-9c4zk" (OuterVolumeSpecName: "kube-api-access-9c4zk") pod "0a160b04-7033-4df7-b78e-1f7759014ae3" (UID: "0a160b04-7033-4df7-b78e-1f7759014ae3"). InnerVolumeSpecName "kube-api-access-9c4zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:23:13.170991 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:13.170899    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a160b04-7033-4df7-b78e-1f7759014ae3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0a160b04-7033-4df7-b78e-1f7759014ae3" (UID: "0a160b04-7033-4df7-b78e-1f7759014ae3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:23:13.235464 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:13.235426    2583 generic.go:356] "Generic (PLEG): container finished" podID="0a160b04-7033-4df7-b78e-1f7759014ae3" containerID="ceb7069f1878573f0204a37d301ecd64977b0e80d2ec38b76e33da9f85a0b0c1" exitCode=0
Jul 02 14:23:13.235730 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:13.235473    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2"
Jul 02 14:23:13.235730 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:13.235487    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2" event={"ID":"0a160b04-7033-4df7-b78e-1f7759014ae3","Type":"ContainerDied","Data":"ceb7069f1878573f0204a37d301ecd64977b0e80d2ec38b76e33da9f85a0b0c1"}
Jul 02 14:23:13.235730 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:13.235535    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2" event={"ID":"0a160b04-7033-4df7-b78e-1f7759014ae3","Type":"ContainerDied","Data":"2013a7b4772cc97d9e848f67bc72436fdeb052f6d1a100fb2284a1de75f76949"}
Jul 02 14:23:13.235730 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:13.235556    2583 scope.go:117] "RemoveContainer" containerID="ceb7069f1878573f0204a37d301ecd64977b0e80d2ec38b76e33da9f85a0b0c1"
Jul 02 14:23:13.246569 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:13.246538    2583 scope.go:117] "RemoveContainer" containerID="ceb7069f1878573f0204a37d301ecd64977b0e80d2ec38b76e33da9f85a0b0c1"
Jul 02 14:23:13.246884 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:23:13.246866    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceb7069f1878573f0204a37d301ecd64977b0e80d2ec38b76e33da9f85a0b0c1\": container with ID starting with ceb7069f1878573f0204a37d301ecd64977b0e80d2ec38b76e33da9f85a0b0c1 not found: ID does not exist" containerID="ceb7069f1878573f0204a37d301ecd64977b0e80d2ec38b76e33da9f85a0b0c1"
Jul 02 14:23:13.246948 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:13.246892    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceb7069f1878573f0204a37d301ecd64977b0e80d2ec38b76e33da9f85a0b0c1"} err="failed to get container status \"ceb7069f1878573f0204a37d301ecd64977b0e80d2ec38b76e33da9f85a0b0c1\": rpc error: code = NotFound desc = could not find container \"ceb7069f1878573f0204a37d301ecd64977b0e80d2ec38b76e33da9f85a0b0c1\": container with ID starting with ceb7069f1878573f0204a37d301ecd64977b0e80d2ec38b76e33da9f85a0b0c1 not found: ID does not exist"
Jul 02 14:23:13.260690 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:13.260650    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2"]
Jul 02 14:23:13.264778 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:13.264747    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87kqrq2"]
Jul 02 14:23:13.270164 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:13.270139    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c4zk\" (UniqueName: \"kubernetes.io/projected/0a160b04-7033-4df7-b78e-1f7759014ae3-kube-api-access-9c4zk\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:23:13.270164 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:13.270165    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a160b04-7033-4df7-b78e-1f7759014ae3-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:23:15.235531 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:15.235494    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a160b04-7033-4df7-b78e-1f7759014ae3" path="/var/lib/kubelet/pods/0a160b04-7033-4df7-b78e-1f7759014ae3/volumes"
Jul 02 14:23:17.975956 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:17.975907    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld" podUID="a22d8d47-875a-4a2b-964a-034eacb7ee77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 02 14:23:27.755668 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:27.755597    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6" podUID="86f61e5c-51e7-4a88-b1fb-9df7aa1bc227" containerName="main" containerID="cri-o://3a532f85a80d18e026d95c92d13484fca2e7c31b00739ab0c55899c0b4d72090" gracePeriod=45
Jul 02 14:23:27.975736 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:27.975697    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld" podUID="a22d8d47-875a-4a2b-964a-034eacb7ee77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 02 14:23:28.035607 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:28.035575    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6"
Jul 02 14:23:28.118232 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:28.118182    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvbsq\" (UniqueName: \"kubernetes.io/projected/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-kube-api-access-nvbsq\") pod \"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227\" (UID: \"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227\") "
Jul 02 14:23:28.118467 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:28.118252    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-model-cache\") pod \"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227\" (UID: \"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227\") "
Jul 02 14:23:28.118467 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:28.118281    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-tls-certs\") pod \"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227\" (UID: \"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227\") "
Jul 02 14:23:28.118467 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:28.118310    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-dshm\") pod \"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227\" (UID: \"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227\") "
Jul 02 14:23:28.118467 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:28.118375    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-tmp-dir\") pod \"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227\" (UID: \"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227\") "
Jul 02 14:23:28.118467 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:28.118428    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-home\") pod \"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227\" (UID: \"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227\") "
Jul 02 14:23:28.119491 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:28.119011    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-model-cache" (OuterVolumeSpecName: "model-cache") pod "86f61e5c-51e7-4a88-b1fb-9df7aa1bc227" (UID: "86f61e5c-51e7-4a88-b1fb-9df7aa1bc227"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:23:28.119491 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:28.119108    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-home" (OuterVolumeSpecName: "home") pod "86f61e5c-51e7-4a88-b1fb-9df7aa1bc227" (UID: "86f61e5c-51e7-4a88-b1fb-9df7aa1bc227"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:23:28.119491 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:28.119454    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "86f61e5c-51e7-4a88-b1fb-9df7aa1bc227" (UID: "86f61e5c-51e7-4a88-b1fb-9df7aa1bc227"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:23:28.121670 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:28.121571    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-kube-api-access-nvbsq" (OuterVolumeSpecName: "kube-api-access-nvbsq") pod "86f61e5c-51e7-4a88-b1fb-9df7aa1bc227" (UID: "86f61e5c-51e7-4a88-b1fb-9df7aa1bc227"). InnerVolumeSpecName "kube-api-access-nvbsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:23:28.121670 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:28.121614    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "86f61e5c-51e7-4a88-b1fb-9df7aa1bc227" (UID: "86f61e5c-51e7-4a88-b1fb-9df7aa1bc227"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:23:28.121950 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:28.121917    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-dshm" (OuterVolumeSpecName: "dshm") pod "86f61e5c-51e7-4a88-b1fb-9df7aa1bc227" (UID: "86f61e5c-51e7-4a88-b1fb-9df7aa1bc227"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:23:28.220147 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:28.220082    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-tmp-dir\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:23:28.220147 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:28.220141    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-home\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:23:28.220147 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:28.220161    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvbsq\" (UniqueName: \"kubernetes.io/projected/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-kube-api-access-nvbsq\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:23:28.220564 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:28.220176    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:23:28.220564 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:28.220192    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:23:28.220564 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:28.220205    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227-dshm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:23:28.336207 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:28.336108    2583 generic.go:356] "Generic (PLEG): container finished" podID="86f61e5c-51e7-4a88-b1fb-9df7aa1bc227" containerID="3a532f85a80d18e026d95c92d13484fca2e7c31b00739ab0c55899c0b4d72090" exitCode=0
Jul 02 14:23:28.336207 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:28.336171    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6" event={"ID":"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227","Type":"ContainerDied","Data":"3a532f85a80d18e026d95c92d13484fca2e7c31b00739ab0c55899c0b4d72090"}
Jul 02 14:23:28.336481 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:28.336221    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6" event={"ID":"86f61e5c-51e7-4a88-b1fb-9df7aa1bc227","Type":"ContainerDied","Data":"df30929e3a8a9c30a057a0ee5f6b7ef664ce62ea2af41945b020340ffbef58ce"}
Jul 02 14:23:28.336481 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:28.336255    2583 scope.go:117] "RemoveContainer" containerID="3a532f85a80d18e026d95c92d13484fca2e7c31b00739ab0c55899c0b4d72090"
Jul 02 14:23:28.336481 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:28.336475    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6"
Jul 02 14:23:28.349826 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:28.349795    2583 scope.go:117] "RemoveContainer" containerID="3a532f85a80d18e026d95c92d13484fca2e7c31b00739ab0c55899c0b4d72090"
Jul 02 14:23:28.350832 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:23:28.350756    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a532f85a80d18e026d95c92d13484fca2e7c31b00739ab0c55899c0b4d72090\": container with ID starting with 3a532f85a80d18e026d95c92d13484fca2e7c31b00739ab0c55899c0b4d72090 not found: ID does not exist" containerID="3a532f85a80d18e026d95c92d13484fca2e7c31b00739ab0c55899c0b4d72090"
Jul 02 14:23:28.350832 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:28.350796    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a532f85a80d18e026d95c92d13484fca2e7c31b00739ab0c55899c0b4d72090"} err="failed to get container status \"3a532f85a80d18e026d95c92d13484fca2e7c31b00739ab0c55899c0b4d72090\": rpc error: code = NotFound desc = could not find container \"3a532f85a80d18e026d95c92d13484fca2e7c31b00739ab0c55899c0b4d72090\": container with ID starting with 3a532f85a80d18e026d95c92d13484fca2e7c31b00739ab0c55899c0b4d72090 not found: ID does not exist"
Jul 02 14:23:28.364622 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:28.364590    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6"]
Jul 02 14:23:28.368336 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:28.368312    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-d56z6"]
Jul 02 14:23:29.236575 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:29.236539    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86f61e5c-51e7-4a88-b1fb-9df7aa1bc227" path="/var/lib/kubelet/pods/86f61e5c-51e7-4a88-b1fb-9df7aa1bc227/volumes"
Jul 02 14:23:37.986142 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:37.986101    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"
Jul 02 14:23:37.994434 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:37.994402    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"
Jul 02 14:23:57.673335 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:57.673298    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:23:57.745064 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:57.745024    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:23:57.800860 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:57.800830    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5"]
Jul 02 14:23:57.801326 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:57.801310    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86f61e5c-51e7-4a88-b1fb-9df7aa1bc227" containerName="main"
Jul 02 14:23:57.801326 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:57.801328    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="86f61e5c-51e7-4a88-b1fb-9df7aa1bc227" containerName="main"
Jul 02 14:23:57.801487 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:57.801337    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a160b04-7033-4df7-b78e-1f7759014ae3" containerName="main"
Jul 02 14:23:57.801487 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:57.801342    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a160b04-7033-4df7-b78e-1f7759014ae3" containerName="main"
Jul 02 14:23:57.801487 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:57.801433    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a160b04-7033-4df7-b78e-1f7759014ae3" containerName="main"
Jul 02 14:23:57.801487 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:57.801446    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="86f61e5c-51e7-4a88-b1fb-9df7aa1bc227" containerName="main"
Jul 02 14:23:57.804263 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:57.804243    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5"
Jul 02 14:23:57.806964 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:57.806944    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 02 14:23:57.813632 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:57.813598    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5"]
Jul 02 14:23:57.903585 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:57.903546    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzbjl\" (UniqueName: \"kubernetes.io/projected/9d5a903c-033a-45c0-9308-4e8340311529-kube-api-access-jzbjl\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5\" (UID: \"9d5a903c-033a-45c0-9308-4e8340311529\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5"
Jul 02 14:23:57.903789 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:57.903603    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d5a903c-033a-45c0-9308-4e8340311529-home\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5\" (UID: \"9d5a903c-033a-45c0-9308-4e8340311529\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5"
Jul 02 14:23:57.903789 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:57.903623    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d5a903c-033a-45c0-9308-4e8340311529-dshm\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5\" (UID: \"9d5a903c-033a-45c0-9308-4e8340311529\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5"
Jul 02 14:23:57.903789 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:57.903690    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d5a903c-033a-45c0-9308-4e8340311529-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5\" (UID: \"9d5a903c-033a-45c0-9308-4e8340311529\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5"
Jul 02 14:23:57.903957 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:57.903791    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d5a903c-033a-45c0-9308-4e8340311529-model-cache\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5\" (UID: \"9d5a903c-033a-45c0-9308-4e8340311529\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5"
Jul 02 14:23:57.903957 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:57.903847    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d5a903c-033a-45c0-9308-4e8340311529-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5\" (UID: \"9d5a903c-033a-45c0-9308-4e8340311529\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5"
Jul 02 14:23:58.005075 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.004982    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzbjl\" (UniqueName: \"kubernetes.io/projected/9d5a903c-033a-45c0-9308-4e8340311529-kube-api-access-jzbjl\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5\" (UID: \"9d5a903c-033a-45c0-9308-4e8340311529\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5"
Jul 02 14:23:58.005075 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.005041    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d5a903c-033a-45c0-9308-4e8340311529-home\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5\" (UID: \"9d5a903c-033a-45c0-9308-4e8340311529\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5"
Jul 02 14:23:58.005075 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.005060    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d5a903c-033a-45c0-9308-4e8340311529-dshm\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5\" (UID: \"9d5a903c-033a-45c0-9308-4e8340311529\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5"
Jul 02 14:23:58.005418 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.005085    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d5a903c-033a-45c0-9308-4e8340311529-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5\" (UID: \"9d5a903c-033a-45c0-9308-4e8340311529\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5"
Jul 02 14:23:58.005418 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.005125    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d5a903c-033a-45c0-9308-4e8340311529-model-cache\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5\" (UID: \"9d5a903c-033a-45c0-9308-4e8340311529\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5"
Jul 02 14:23:58.005418 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.005149    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d5a903c-033a-45c0-9308-4e8340311529-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5\" (UID: \"9d5a903c-033a-45c0-9308-4e8340311529\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5"
Jul 02 14:23:58.005618 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.005592    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d5a903c-033a-45c0-9308-4e8340311529-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5\" (UID: \"9d5a903c-033a-45c0-9308-4e8340311529\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5"
Jul 02 14:23:58.005618 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.005609    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d5a903c-033a-45c0-9308-4e8340311529-model-cache\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5\" (UID: \"9d5a903c-033a-45c0-9308-4e8340311529\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5"
Jul 02 14:23:58.005725 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.005653    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d5a903c-033a-45c0-9308-4e8340311529-home\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5\" (UID: \"9d5a903c-033a-45c0-9308-4e8340311529\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5"
Jul 02 14:23:58.007378 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.007341    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d5a903c-033a-45c0-9308-4e8340311529-dshm\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5\" (UID: \"9d5a903c-033a-45c0-9308-4e8340311529\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5"
Jul 02 14:23:58.007636 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.007618    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d5a903c-033a-45c0-9308-4e8340311529-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5\" (UID: \"9d5a903c-033a-45c0-9308-4e8340311529\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5"
Jul 02 14:23:58.014149 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.014123    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzbjl\" (UniqueName: \"kubernetes.io/projected/9d5a903c-033a-45c0-9308-4e8340311529-kube-api-access-jzbjl\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5\" (UID: \"9d5a903c-033a-45c0-9308-4e8340311529\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5"
Jul 02 14:23:58.079310 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.079269    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:23:58.112118 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.112088    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465"]
Jul 02 14:23:58.115105 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.115083    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465"
Jul 02 14:23:58.116169 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.116147    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5"
Jul 02 14:23:58.117699 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.117675    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-c54fj\""
Jul 02 14:23:58.127455 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.127431    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465"]
Jul 02 14:23:58.177080 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.177028    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:23:58.208083 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.208028    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64af7bf4-19ad-422e-9a29-684bad7281b0-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465\" (UID: \"64af7bf4-19ad-422e-9a29-684bad7281b0\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465"
Jul 02 14:23:58.208225 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.208202    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/64af7bf4-19ad-422e-9a29-684bad7281b0-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465\" (UID: \"64af7bf4-19ad-422e-9a29-684bad7281b0\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465"
Jul 02 14:23:58.208287 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.208259    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnjl5\" (UniqueName: \"kubernetes.io/projected/64af7bf4-19ad-422e-9a29-684bad7281b0-kube-api-access-nnjl5\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465\" (UID: \"64af7bf4-19ad-422e-9a29-684bad7281b0\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465"
Jul 02 14:23:58.261026 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.260992    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5"]
Jul 02 14:23:58.263337 ip-10-0-135-143 kubenswrapper[2583]: W0702 14:23:58.263307    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5a903c_033a_45c0_9308_4e8340311529.slice/crio-697ebe9cbe96fdc560d59adfd9a56d9ff0e30748324ea299afc9ebc7f361d5cb WatchSource:0}: Error finding container 697ebe9cbe96fdc560d59adfd9a56d9ff0e30748324ea299afc9ebc7f361d5cb: Status 404 returned error can't find the container with id 697ebe9cbe96fdc560d59adfd9a56d9ff0e30748324ea299afc9ebc7f361d5cb
Jul 02 14:23:58.308960 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.308863    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64af7bf4-19ad-422e-9a29-684bad7281b0-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465\" (UID: \"64af7bf4-19ad-422e-9a29-684bad7281b0\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465"
Jul 02 14:23:58.309053 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.308972    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/64af7bf4-19ad-422e-9a29-684bad7281b0-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465\" (UID: \"64af7bf4-19ad-422e-9a29-684bad7281b0\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465"
Jul 02 14:23:58.309053 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.309022    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnjl5\" (UniqueName: \"kubernetes.io/projected/64af7bf4-19ad-422e-9a29-684bad7281b0-kube-api-access-nnjl5\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465\" (UID: \"64af7bf4-19ad-422e-9a29-684bad7281b0\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465"
Jul 02 14:23:58.309656 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.309630    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/64af7bf4-19ad-422e-9a29-684bad7281b0-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465\" (UID: \"64af7bf4-19ad-422e-9a29-684bad7281b0\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465"
Jul 02 14:23:58.311561 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.311531    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64af7bf4-19ad-422e-9a29-684bad7281b0-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465\" (UID: \"64af7bf4-19ad-422e-9a29-684bad7281b0\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465"
Jul 02 14:23:58.318691 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.318665    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnjl5\" (UniqueName: \"kubernetes.io/projected/64af7bf4-19ad-422e-9a29-684bad7281b0-kube-api-access-nnjl5\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465\" (UID: \"64af7bf4-19ad-422e-9a29-684bad7281b0\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465"
Jul 02 14:23:58.427287 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.427249    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465"
Jul 02 14:23:58.491587 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.491554    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5" event={"ID":"9d5a903c-033a-45c0-9308-4e8340311529","Type":"ContainerStarted","Data":"573e15191659ab20106818bf0c2c2e383051b119dacca832aaf9473e0432b2f4"}
Jul 02 14:23:58.491744 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.491609    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5" event={"ID":"9d5a903c-033a-45c0-9308-4e8340311529","Type":"ContainerStarted","Data":"697ebe9cbe96fdc560d59adfd9a56d9ff0e30748324ea299afc9ebc7f361d5cb"}
Jul 02 14:23:58.509486 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.509428    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5" podStartSLOduration=1.50940714 podStartE2EDuration="1.50940714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:23:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:23:58.508979937 +0000 UTC m=+4149.736891111" watchObservedRunningTime="2026-07-02 14:23:58.50940714 +0000 UTC m=+4149.737318320"
Jul 02 14:23:58.570869 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:58.570836    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465"]
Jul 02 14:23:58.572745 ip-10-0-135-143 kubenswrapper[2583]: W0702 14:23:58.572717    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64af7bf4_19ad_422e_9a29_684bad7281b0.slice/crio-64d6c221cce92145b431c5f6e497a9d9f6da52034205034019eea386aa8fc662 WatchSource:0}: Error finding container 64d6c221cce92145b431c5f6e497a9d9f6da52034205034019eea386aa8fc662: Status 404 returned error can't find the container with id 64d6c221cce92145b431c5f6e497a9d9f6da52034205034019eea386aa8fc662
Jul 02 14:23:59.116640 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:59.116591    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5"
Jul 02 14:23:59.501460 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:59.501340    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465" event={"ID":"64af7bf4-19ad-422e-9a29-684bad7281b0","Type":"ContainerStarted","Data":"c2a8f53a39ed8c99032a351ab4e6c41befe8211f616f31943ca28426b2cff05d"}
Jul 02 14:23:59.501460 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:59.501418    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465" event={"ID":"64af7bf4-19ad-422e-9a29-684bad7281b0","Type":"ContainerStarted","Data":"64d6c221cce92145b431c5f6e497a9d9f6da52034205034019eea386aa8fc662"}
Jul 02 14:23:59.501715 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:59.501677    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465"
Jul 02 14:23:59.518763 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:23:59.518711    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465" podStartSLOduration=1.518693872 podStartE2EDuration="1.518693872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:23:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:23:59.517568924 +0000 UTC m=+4150.745480099" watchObservedRunningTime="2026-07-02 14:23:59.518693872 +0000 UTC m=+4150.746605049"
Jul 02 14:24:08.116769 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:08.116723    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5"
Jul 02 14:24:08.129194 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:08.129163    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5"
Jul 02 14:24:08.556771 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:08.556684    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5"
Jul 02 14:24:30.507474 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:30.507441    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465"
Jul 02 14:24:31.988534 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:31.988491    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:24:46.882497 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:46.882461    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"]
Jul 02 14:24:46.895247 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:46.895209    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:24:49.634961 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:49.634931    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 14:24:49.646592 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:49.646567    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 14:24:49.911719 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:49.911633    2583 scope.go:117] "RemoveContainer" containerID="e9e0271bf66f50b666693a38296c2eb1f7921e8417f9151bd12ad9245242fdc7"
Jul 02 14:24:49.920832 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:49.920804    2583 scope.go:117] "RemoveContainer" containerID="a5d5bcbf9ef03b48de1dd76bf4dfb6aa7c50cd077758e2d833289f2a27eac1b4"
Jul 02 14:24:53.011172 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:53.011134    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465"]
Jul 02 14:24:53.019306 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:53.019273    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5"]
Jul 02 14:24:53.022483 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:24:53.022434    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 02 14:24:53.022483 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 14:24:53.022483 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 14:24:53.022483 ip-10-0-135-143 kubenswrapper[2583]:  > containerID="c2a8f53a39ed8c99032a351ab4e6c41befe8211f616f31943ca28426b2cff05d" cmd=["/bin/sleep","15"]
Jul 02 14:24:53.022772 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:53.022506    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465" message="" err=<
Jul 02 14:24:53.022772 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 14:24:53.022772 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 14:24:53.022772 ip-10-0-135-143 kubenswrapper[2583]:  >
Jul 02 14:24:53.022772 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:24:53.022527    2583 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 02 14:24:53.022772 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 14:24:53.022772 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 14:24:53.022772 ip-10-0-135-143 kubenswrapper[2583]:  > pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465" podUID="64af7bf4-19ad-422e-9a29-684bad7281b0" containerName="main" containerID="cri-o://c2a8f53a39ed8c99032a351ab4e6c41befe8211f616f31943ca28426b2cff05d"
Jul 02 14:24:53.022772 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:53.022577    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465" podUID="64af7bf4-19ad-422e-9a29-684bad7281b0" containerName="main" containerID="cri-o://c2a8f53a39ed8c99032a351ab4e6c41befe8211f616f31943ca28426b2cff05d" gracePeriod=60
Jul 02 14:24:53.039628 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:53.039595    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:24:53.122672 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:53.122628    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:24:53.278318 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:53.278293    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465"
Jul 02 14:24:53.374122 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:53.374085    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnjl5\" (UniqueName: \"kubernetes.io/projected/64af7bf4-19ad-422e-9a29-684bad7281b0-kube-api-access-nnjl5\") pod \"64af7bf4-19ad-422e-9a29-684bad7281b0\" (UID: \"64af7bf4-19ad-422e-9a29-684bad7281b0\") "
Jul 02 14:24:53.374326 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:53.374159    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64af7bf4-19ad-422e-9a29-684bad7281b0-tls-certs\") pod \"64af7bf4-19ad-422e-9a29-684bad7281b0\" (UID: \"64af7bf4-19ad-422e-9a29-684bad7281b0\") "
Jul 02 14:24:53.374326 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:53.374235    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/64af7bf4-19ad-422e-9a29-684bad7281b0-tokenizer-uds\") pod \"64af7bf4-19ad-422e-9a29-684bad7281b0\" (UID: \"64af7bf4-19ad-422e-9a29-684bad7281b0\") "
Jul 02 14:24:53.374562 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:53.374537    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64af7bf4-19ad-422e-9a29-684bad7281b0-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "64af7bf4-19ad-422e-9a29-684bad7281b0" (UID: "64af7bf4-19ad-422e-9a29-684bad7281b0"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:24:53.374658 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:53.374639    2583 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/64af7bf4-19ad-422e-9a29-684bad7281b0-tokenizer-uds\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:24:53.376347 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:53.376320    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64af7bf4-19ad-422e-9a29-684bad7281b0-kube-api-access-nnjl5" (OuterVolumeSpecName: "kube-api-access-nnjl5") pod "64af7bf4-19ad-422e-9a29-684bad7281b0" (UID: "64af7bf4-19ad-422e-9a29-684bad7281b0"). InnerVolumeSpecName "kube-api-access-nnjl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:24:53.376441 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:53.376326    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64af7bf4-19ad-422e-9a29-684bad7281b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "64af7bf4-19ad-422e-9a29-684bad7281b0" (UID: "64af7bf4-19ad-422e-9a29-684bad7281b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:24:53.475351 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:53.475312    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnjl5\" (UniqueName: \"kubernetes.io/projected/64af7bf4-19ad-422e-9a29-684bad7281b0-kube-api-access-nnjl5\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:24:53.475351 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:53.475343    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64af7bf4-19ad-422e-9a29-684bad7281b0-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:24:53.804252 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:53.804216    2583 generic.go:356] "Generic (PLEG): container finished" podID="64af7bf4-19ad-422e-9a29-684bad7281b0" containerID="c2a8f53a39ed8c99032a351ab4e6c41befe8211f616f31943ca28426b2cff05d" exitCode=0
Jul 02 14:24:53.804477 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:53.804265    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465"
Jul 02 14:24:53.804477 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:53.804309    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465" event={"ID":"64af7bf4-19ad-422e-9a29-684bad7281b0","Type":"ContainerDied","Data":"c2a8f53a39ed8c99032a351ab4e6c41befe8211f616f31943ca28426b2cff05d"}
Jul 02 14:24:53.804477 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:53.804362    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465" event={"ID":"64af7bf4-19ad-422e-9a29-684bad7281b0","Type":"ContainerDied","Data":"64d6c221cce92145b431c5f6e497a9d9f6da52034205034019eea386aa8fc662"}
Jul 02 14:24:53.804477 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:53.804383    2583 scope.go:117] "RemoveContainer" containerID="c2a8f53a39ed8c99032a351ab4e6c41befe8211f616f31943ca28426b2cff05d"
Jul 02 14:24:53.813757 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:53.813738    2583 scope.go:117] "RemoveContainer" containerID="c2a8f53a39ed8c99032a351ab4e6c41befe8211f616f31943ca28426b2cff05d"
Jul 02 14:24:53.814045 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:24:53.814025    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2a8f53a39ed8c99032a351ab4e6c41befe8211f616f31943ca28426b2cff05d\": container with ID starting with c2a8f53a39ed8c99032a351ab4e6c41befe8211f616f31943ca28426b2cff05d not found: ID does not exist" containerID="c2a8f53a39ed8c99032a351ab4e6c41befe8211f616f31943ca28426b2cff05d"
Jul 02 14:24:53.814107 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:53.814058    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2a8f53a39ed8c99032a351ab4e6c41befe8211f616f31943ca28426b2cff05d"} err="failed to get container status \"c2a8f53a39ed8c99032a351ab4e6c41befe8211f616f31943ca28426b2cff05d\": rpc error: code = NotFound desc = could not find container \"c2a8f53a39ed8c99032a351ab4e6c41befe8211f616f31943ca28426b2cff05d\": container with ID starting with c2a8f53a39ed8c99032a351ab4e6c41befe8211f616f31943ca28426b2cff05d not found: ID does not exist"
Jul 02 14:24:53.827572 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:53.827545    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465"]
Jul 02 14:24:53.831535 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:53.831512    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d77465"]
Jul 02 14:24:55.233502 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:24:55.233468    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64af7bf4-19ad-422e-9a29-684bad7281b0" path="/var/lib/kubelet/pods/64af7bf4-19ad-422e-9a29-684bad7281b0/volumes"
Jul 02 14:25:01.906255 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:01.906207    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld" podUID="a22d8d47-875a-4a2b-964a-034eacb7ee77" containerName="main" containerID="cri-o://336cd154becf8ead8d1b26ea9c517e1edfb5fbde6b86b8e446fc4b3e7b21d8e4" gracePeriod=45
Jul 02 14:25:01.987536 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:01.987494    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld" podUID="a22d8d47-875a-4a2b-964a-034eacb7ee77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 02 14:25:02.575841 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.575813    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"
Jul 02 14:25:02.663364 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.663277    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gr5lk\" (UniqueName: \"kubernetes.io/projected/a22d8d47-875a-4a2b-964a-034eacb7ee77-kube-api-access-gr5lk\") pod \"a22d8d47-875a-4a2b-964a-034eacb7ee77\" (UID: \"a22d8d47-875a-4a2b-964a-034eacb7ee77\") "
Jul 02 14:25:02.663364 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.663343    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22d8d47-875a-4a2b-964a-034eacb7ee77-tls-certs\") pod \"a22d8d47-875a-4a2b-964a-034eacb7ee77\" (UID: \"a22d8d47-875a-4a2b-964a-034eacb7ee77\") "
Jul 02 14:25:02.663644 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.663376    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22d8d47-875a-4a2b-964a-034eacb7ee77-home\") pod \"a22d8d47-875a-4a2b-964a-034eacb7ee77\" (UID: \"a22d8d47-875a-4a2b-964a-034eacb7ee77\") "
Jul 02 14:25:02.663644 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.663495    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a22d8d47-875a-4a2b-964a-034eacb7ee77-cabundle-cert\") pod \"a22d8d47-875a-4a2b-964a-034eacb7ee77\" (UID: \"a22d8d47-875a-4a2b-964a-034eacb7ee77\") "
Jul 02 14:25:02.663644 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.663532    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a22d8d47-875a-4a2b-964a-034eacb7ee77-kserve-provision-location\") pod \"a22d8d47-875a-4a2b-964a-034eacb7ee77\" (UID: \"a22d8d47-875a-4a2b-964a-034eacb7ee77\") "
Jul 02 14:25:02.663805 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.663752    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22d8d47-875a-4a2b-964a-034eacb7ee77-tmp-dir\") pod \"a22d8d47-875a-4a2b-964a-034eacb7ee77\" (UID: \"a22d8d47-875a-4a2b-964a-034eacb7ee77\") "
Jul 02 14:25:02.663859 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.663801    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22d8d47-875a-4a2b-964a-034eacb7ee77-model-cache\") pod \"a22d8d47-875a-4a2b-964a-034eacb7ee77\" (UID: \"a22d8d47-875a-4a2b-964a-034eacb7ee77\") "
Jul 02 14:25:02.663859 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.663844    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22d8d47-875a-4a2b-964a-034eacb7ee77-dshm\") pod \"a22d8d47-875a-4a2b-964a-034eacb7ee77\" (UID: \"a22d8d47-875a-4a2b-964a-034eacb7ee77\") "
Jul 02 14:25:02.663967 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.663889    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22d8d47-875a-4a2b-964a-034eacb7ee77-home" (OuterVolumeSpecName: "home") pod "a22d8d47-875a-4a2b-964a-034eacb7ee77" (UID: "a22d8d47-875a-4a2b-964a-034eacb7ee77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:25:02.663967 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.663903    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a22d8d47-875a-4a2b-964a-034eacb7ee77-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a22d8d47-875a-4a2b-964a-034eacb7ee77" (UID: "a22d8d47-875a-4a2b-964a-034eacb7ee77"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:25:02.664200 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.664156    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22d8d47-875a-4a2b-964a-034eacb7ee77-model-cache" (OuterVolumeSpecName: "model-cache") pod "a22d8d47-875a-4a2b-964a-034eacb7ee77" (UID: "a22d8d47-875a-4a2b-964a-034eacb7ee77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:25:02.664307 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.664220    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a22d8d47-875a-4a2b-964a-034eacb7ee77-cabundle-cert\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:25:02.664307 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.664242    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22d8d47-875a-4a2b-964a-034eacb7ee77-home\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:25:02.665867 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.665836    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a22d8d47-875a-4a2b-964a-034eacb7ee77-kube-api-access-gr5lk" (OuterVolumeSpecName: "kube-api-access-gr5lk") pod "a22d8d47-875a-4a2b-964a-034eacb7ee77" (UID: "a22d8d47-875a-4a2b-964a-034eacb7ee77"). InnerVolumeSpecName "kube-api-access-gr5lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:25:02.665984 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.665919    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a22d8d47-875a-4a2b-964a-034eacb7ee77-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a22d8d47-875a-4a2b-964a-034eacb7ee77" (UID: "a22d8d47-875a-4a2b-964a-034eacb7ee77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:25:02.666454 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.666428    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22d8d47-875a-4a2b-964a-034eacb7ee77-dshm" (OuterVolumeSpecName: "dshm") pod "a22d8d47-875a-4a2b-964a-034eacb7ee77" (UID: "a22d8d47-875a-4a2b-964a-034eacb7ee77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:25:02.675509 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.675473    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22d8d47-875a-4a2b-964a-034eacb7ee77-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a22d8d47-875a-4a2b-964a-034eacb7ee77" (UID: "a22d8d47-875a-4a2b-964a-034eacb7ee77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:25:02.729766 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.729720    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22d8d47-875a-4a2b-964a-034eacb7ee77-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a22d8d47-875a-4a2b-964a-034eacb7ee77" (UID: "a22d8d47-875a-4a2b-964a-034eacb7ee77"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:25:02.765579 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.765534    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr5lk\" (UniqueName: \"kubernetes.io/projected/a22d8d47-875a-4a2b-964a-034eacb7ee77-kube-api-access-gr5lk\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:25:02.765579 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.765567    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22d8d47-875a-4a2b-964a-034eacb7ee77-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:25:02.765579 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.765577    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a22d8d47-875a-4a2b-964a-034eacb7ee77-kserve-provision-location\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:25:02.765579 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.765588    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22d8d47-875a-4a2b-964a-034eacb7ee77-tmp-dir\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:25:02.765841 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.765597    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22d8d47-875a-4a2b-964a-034eacb7ee77-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:25:02.765841 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.765605    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22d8d47-875a-4a2b-964a-034eacb7ee77-dshm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:25:02.851549 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.851517    2583 generic.go:356] "Generic (PLEG): container finished" podID="a22d8d47-875a-4a2b-964a-034eacb7ee77" containerID="336cd154becf8ead8d1b26ea9c517e1edfb5fbde6b86b8e446fc4b3e7b21d8e4" exitCode=0
Jul 02 14:25:02.851744 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.851578    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"
Jul 02 14:25:02.851744 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.851612    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld" event={"ID":"a22d8d47-875a-4a2b-964a-034eacb7ee77","Type":"ContainerDied","Data":"336cd154becf8ead8d1b26ea9c517e1edfb5fbde6b86b8e446fc4b3e7b21d8e4"}
Jul 02 14:25:02.851744 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.851701    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld" event={"ID":"a22d8d47-875a-4a2b-964a-034eacb7ee77","Type":"ContainerDied","Data":"5a90934a6f97a83bfd81d11bd2af18ebb654dccce79332b01bca85382ca55dca"}
Jul 02 14:25:02.851744 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.851723    2583 scope.go:117] "RemoveContainer" containerID="336cd154becf8ead8d1b26ea9c517e1edfb5fbde6b86b8e446fc4b3e7b21d8e4"
Jul 02 14:25:02.861025 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.861005    2583 scope.go:117] "RemoveContainer" containerID="f4b480bd2f479aa033ea405a82a509f828f23ef3ec77dd8149d007a2911870d5"
Jul 02 14:25:02.869707 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.869687    2583 scope.go:117] "RemoveContainer" containerID="336cd154becf8ead8d1b26ea9c517e1edfb5fbde6b86b8e446fc4b3e7b21d8e4"
Jul 02 14:25:02.869992 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:25:02.869972    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"336cd154becf8ead8d1b26ea9c517e1edfb5fbde6b86b8e446fc4b3e7b21d8e4\": container with ID starting with 336cd154becf8ead8d1b26ea9c517e1edfb5fbde6b86b8e446fc4b3e7b21d8e4 not found: ID does not exist" containerID="336cd154becf8ead8d1b26ea9c517e1edfb5fbde6b86b8e446fc4b3e7b21d8e4"
Jul 02 14:25:02.870043 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.870003    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"336cd154becf8ead8d1b26ea9c517e1edfb5fbde6b86b8e446fc4b3e7b21d8e4"} err="failed to get container status \"336cd154becf8ead8d1b26ea9c517e1edfb5fbde6b86b8e446fc4b3e7b21d8e4\": rpc error: code = NotFound desc = could not find container \"336cd154becf8ead8d1b26ea9c517e1edfb5fbde6b86b8e446fc4b3e7b21d8e4\": container with ID starting with 336cd154becf8ead8d1b26ea9c517e1edfb5fbde6b86b8e446fc4b3e7b21d8e4 not found: ID does not exist"
Jul 02 14:25:02.870043 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.870021    2583 scope.go:117] "RemoveContainer" containerID="f4b480bd2f479aa033ea405a82a509f828f23ef3ec77dd8149d007a2911870d5"
Jul 02 14:25:02.870265 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:25:02.870251    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b480bd2f479aa033ea405a82a509f828f23ef3ec77dd8149d007a2911870d5\": container with ID starting with f4b480bd2f479aa033ea405a82a509f828f23ef3ec77dd8149d007a2911870d5 not found: ID does not exist" containerID="f4b480bd2f479aa033ea405a82a509f828f23ef3ec77dd8149d007a2911870d5"
Jul 02 14:25:02.870307 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.870269    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b480bd2f479aa033ea405a82a509f828f23ef3ec77dd8149d007a2911870d5"} err="failed to get container status \"f4b480bd2f479aa033ea405a82a509f828f23ef3ec77dd8149d007a2911870d5\": rpc error: code = NotFound desc = could not find container \"f4b480bd2f479aa033ea405a82a509f828f23ef3ec77dd8149d007a2911870d5\": container with ID starting with f4b480bd2f479aa033ea405a82a509f828f23ef3ec77dd8149d007a2911870d5 not found: ID does not exist"
Jul 02 14:25:02.874768 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.874739    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"]
Jul 02 14:25:02.878536 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:02.878512    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-6cfb96b7d7-g29ld"]
Jul 02 14:25:03.237136 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:03.237083    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a22d8d47-875a-4a2b-964a-034eacb7ee77" path="/var/lib/kubelet/pods/a22d8d47-875a-4a2b-964a-034eacb7ee77/volumes"
Jul 02 14:25:08.042774 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:08.042723    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5" podUID="9d5a903c-033a-45c0-9308-4e8340311529" containerName="main" containerID="cri-o://573e15191659ab20106818bf0c2c2e383051b119dacca832aaf9473e0432b2f4" gracePeriod=45
Jul 02 14:25:08.297234 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:08.297168    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5"
Jul 02 14:25:08.415457 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:08.415417    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d5a903c-033a-45c0-9308-4e8340311529-tls-certs\") pod \"9d5a903c-033a-45c0-9308-4e8340311529\" (UID: \"9d5a903c-033a-45c0-9308-4e8340311529\") "
Jul 02 14:25:08.415658 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:08.415516    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d5a903c-033a-45c0-9308-4e8340311529-model-cache\") pod \"9d5a903c-033a-45c0-9308-4e8340311529\" (UID: \"9d5a903c-033a-45c0-9308-4e8340311529\") "
Jul 02 14:25:08.415658 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:08.415582    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d5a903c-033a-45c0-9308-4e8340311529-home\") pod \"9d5a903c-033a-45c0-9308-4e8340311529\" (UID: \"9d5a903c-033a-45c0-9308-4e8340311529\") "
Jul 02 14:25:08.415658 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:08.415615    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d5a903c-033a-45c0-9308-4e8340311529-tmp-dir\") pod \"9d5a903c-033a-45c0-9308-4e8340311529\" (UID: \"9d5a903c-033a-45c0-9308-4e8340311529\") "
Jul 02 14:25:08.415658 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:08.415652    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d5a903c-033a-45c0-9308-4e8340311529-dshm\") pod \"9d5a903c-033a-45c0-9308-4e8340311529\" (UID: \"9d5a903c-033a-45c0-9308-4e8340311529\") "
Jul 02 14:25:08.415860 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:08.415681    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzbjl\" (UniqueName: \"kubernetes.io/projected/9d5a903c-033a-45c0-9308-4e8340311529-kube-api-access-jzbjl\") pod \"9d5a903c-033a-45c0-9308-4e8340311529\" (UID: \"9d5a903c-033a-45c0-9308-4e8340311529\") "
Jul 02 14:25:08.415860 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:08.415785    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d5a903c-033a-45c0-9308-4e8340311529-model-cache" (OuterVolumeSpecName: "model-cache") pod "9d5a903c-033a-45c0-9308-4e8340311529" (UID: "9d5a903c-033a-45c0-9308-4e8340311529"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:25:08.415978 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:08.415874    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d5a903c-033a-45c0-9308-4e8340311529-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9d5a903c-033a-45c0-9308-4e8340311529" (UID: "9d5a903c-033a-45c0-9308-4e8340311529"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:25:08.415978 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:08.415901    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d5a903c-033a-45c0-9308-4e8340311529-home" (OuterVolumeSpecName: "home") pod "9d5a903c-033a-45c0-9308-4e8340311529" (UID: "9d5a903c-033a-45c0-9308-4e8340311529"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:25:08.416087 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:08.416071    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d5a903c-033a-45c0-9308-4e8340311529-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:25:08.416127 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:08.416091    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d5a903c-033a-45c0-9308-4e8340311529-home\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:25:08.416127 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:08.416116    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d5a903c-033a-45c0-9308-4e8340311529-tmp-dir\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:25:08.417679 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:08.417647    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d5a903c-033a-45c0-9308-4e8340311529-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9d5a903c-033a-45c0-9308-4e8340311529" (UID: "9d5a903c-033a-45c0-9308-4e8340311529"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:25:08.417823 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:08.417801    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d5a903c-033a-45c0-9308-4e8340311529-dshm" (OuterVolumeSpecName: "dshm") pod "9d5a903c-033a-45c0-9308-4e8340311529" (UID: "9d5a903c-033a-45c0-9308-4e8340311529"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:25:08.417876 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:08.417815    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d5a903c-033a-45c0-9308-4e8340311529-kube-api-access-jzbjl" (OuterVolumeSpecName: "kube-api-access-jzbjl") pod "9d5a903c-033a-45c0-9308-4e8340311529" (UID: "9d5a903c-033a-45c0-9308-4e8340311529"). InnerVolumeSpecName "kube-api-access-jzbjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:25:08.517529 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:08.517484    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d5a903c-033a-45c0-9308-4e8340311529-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:25:08.517529 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:08.517514    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d5a903c-033a-45c0-9308-4e8340311529-dshm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:25:08.517529 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:08.517529    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzbjl\" (UniqueName: \"kubernetes.io/projected/9d5a903c-033a-45c0-9308-4e8340311529-kube-api-access-jzbjl\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:25:08.884347 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:08.884306    2583 generic.go:356] "Generic (PLEG): container finished" podID="9d5a903c-033a-45c0-9308-4e8340311529" containerID="573e15191659ab20106818bf0c2c2e383051b119dacca832aaf9473e0432b2f4" exitCode=0
Jul 02 14:25:08.884542 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:08.884354    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5"
Jul 02 14:25:08.884542 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:08.884379    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5" event={"ID":"9d5a903c-033a-45c0-9308-4e8340311529","Type":"ContainerDied","Data":"573e15191659ab20106818bf0c2c2e383051b119dacca832aaf9473e0432b2f4"}
Jul 02 14:25:08.884542 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:08.884446    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5" event={"ID":"9d5a903c-033a-45c0-9308-4e8340311529","Type":"ContainerDied","Data":"697ebe9cbe96fdc560d59adfd9a56d9ff0e30748324ea299afc9ebc7f361d5cb"}
Jul 02 14:25:08.884542 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:08.884468    2583 scope.go:117] "RemoveContainer" containerID="573e15191659ab20106818bf0c2c2e383051b119dacca832aaf9473e0432b2f4"
Jul 02 14:25:08.893583 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:08.893561    2583 scope.go:117] "RemoveContainer" containerID="573e15191659ab20106818bf0c2c2e383051b119dacca832aaf9473e0432b2f4"
Jul 02 14:25:08.893885 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:25:08.893863    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"573e15191659ab20106818bf0c2c2e383051b119dacca832aaf9473e0432b2f4\": container with ID starting with 573e15191659ab20106818bf0c2c2e383051b119dacca832aaf9473e0432b2f4 not found: ID does not exist" containerID="573e15191659ab20106818bf0c2c2e383051b119dacca832aaf9473e0432b2f4"
Jul 02 14:25:08.893953 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:08.893894    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"573e15191659ab20106818bf0c2c2e383051b119dacca832aaf9473e0432b2f4"} err="failed to get container status \"573e15191659ab20106818bf0c2c2e383051b119dacca832aaf9473e0432b2f4\": rpc error: code = NotFound desc = could not find container \"573e15191659ab20106818bf0c2c2e383051b119dacca832aaf9473e0432b2f4\": container with ID starting with 573e15191659ab20106818bf0c2c2e383051b119dacca832aaf9473e0432b2f4 not found: ID does not exist"
Jul 02 14:25:08.907926 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:08.907896    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5"]
Jul 02 14:25:08.912667 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:08.912634    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-xtvt5"]
Jul 02 14:25:09.233576 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:09.233476    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d5a903c-033a-45c0-9308-4e8340311529" path="/var/lib/kubelet/pods/9d5a903c-033a-45c0-9308-4e8340311529/volumes"
Jul 02 14:25:34.059165 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.059083    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"]
Jul 02 14:25:34.059661 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.059462    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64af7bf4-19ad-422e-9a29-684bad7281b0" containerName="main"
Jul 02 14:25:34.059661 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.059474    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="64af7bf4-19ad-422e-9a29-684bad7281b0" containerName="main"
Jul 02 14:25:34.059661 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.059487    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d5a903c-033a-45c0-9308-4e8340311529" containerName="main"
Jul 02 14:25:34.059661 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.059492    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d5a903c-033a-45c0-9308-4e8340311529" containerName="main"
Jul 02 14:25:34.059661 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.059507    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a22d8d47-875a-4a2b-964a-034eacb7ee77" containerName="storage-initializer"
Jul 02 14:25:34.059661 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.059512    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a22d8d47-875a-4a2b-964a-034eacb7ee77" containerName="storage-initializer"
Jul 02 14:25:34.059661 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.059519    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a22d8d47-875a-4a2b-964a-034eacb7ee77" containerName="main"
Jul 02 14:25:34.059661 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.059524    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a22d8d47-875a-4a2b-964a-034eacb7ee77" containerName="main"
Jul 02 14:25:34.059661 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.059584    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d5a903c-033a-45c0-9308-4e8340311529" containerName="main"
Jul 02 14:25:34.059661 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.059592    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="64af7bf4-19ad-422e-9a29-684bad7281b0" containerName="main"
Jul 02 14:25:34.059661 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.059601    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a22d8d47-875a-4a2b-964a-034eacb7ee77" containerName="main"
Jul 02 14:25:34.064371 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.064352    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"
Jul 02 14:25:34.068584 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.068560    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 14:25:34.068909 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.068894    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 14:25:34.069225 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.069207    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Jul 02 14:25:34.084546 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.084516    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"]
Jul 02 14:25:34.100760 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.100726    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:25:34.148379 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.148335    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/131a76e2-7e0d-4225-84f2-6f7e2e445407-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-76c48f974d-4rrrn\" (UID: \"131a76e2-7e0d-4225-84f2-6f7e2e445407\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"
Jul 02 14:25:34.148608 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.148423    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/131a76e2-7e0d-4225-84f2-6f7e2e445407-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-76c48f974d-4rrrn\" (UID: \"131a76e2-7e0d-4225-84f2-6f7e2e445407\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"
Jul 02 14:25:34.148608 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.148455    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/131a76e2-7e0d-4225-84f2-6f7e2e445407-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-76c48f974d-4rrrn\" (UID: \"131a76e2-7e0d-4225-84f2-6f7e2e445407\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"
Jul 02 14:25:34.148608 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.148506    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/131a76e2-7e0d-4225-84f2-6f7e2e445407-model-cache\") pod \"lora-multiple-adapters-test-kserve-76c48f974d-4rrrn\" (UID: \"131a76e2-7e0d-4225-84f2-6f7e2e445407\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"
Jul 02 14:25:34.148608 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.148588    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk25m\" (UniqueName: \"kubernetes.io/projected/131a76e2-7e0d-4225-84f2-6f7e2e445407-kube-api-access-nk25m\") pod \"lora-multiple-adapters-test-kserve-76c48f974d-4rrrn\" (UID: \"131a76e2-7e0d-4225-84f2-6f7e2e445407\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"
Jul 02 14:25:34.148791 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.148627    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/131a76e2-7e0d-4225-84f2-6f7e2e445407-home\") pod \"lora-multiple-adapters-test-kserve-76c48f974d-4rrrn\" (UID: \"131a76e2-7e0d-4225-84f2-6f7e2e445407\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"
Jul 02 14:25:34.148791 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.148644    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/131a76e2-7e0d-4225-84f2-6f7e2e445407-dshm\") pod \"lora-multiple-adapters-test-kserve-76c48f974d-4rrrn\" (UID: \"131a76e2-7e0d-4225-84f2-6f7e2e445407\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"
Jul 02 14:25:34.148791 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.148681    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/131a76e2-7e0d-4225-84f2-6f7e2e445407-tls-certs\") pod \"lora-multiple-adapters-test-kserve-76c48f974d-4rrrn\" (UID: \"131a76e2-7e0d-4225-84f2-6f7e2e445407\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"
Jul 02 14:25:34.249956 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.249913    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/131a76e2-7e0d-4225-84f2-6f7e2e445407-tls-certs\") pod \"lora-multiple-adapters-test-kserve-76c48f974d-4rrrn\" (UID: \"131a76e2-7e0d-4225-84f2-6f7e2e445407\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"
Jul 02 14:25:34.249956 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.249962    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/131a76e2-7e0d-4225-84f2-6f7e2e445407-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-76c48f974d-4rrrn\" (UID: \"131a76e2-7e0d-4225-84f2-6f7e2e445407\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"
Jul 02 14:25:34.250202 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.250086    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/131a76e2-7e0d-4225-84f2-6f7e2e445407-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-76c48f974d-4rrrn\" (UID: \"131a76e2-7e0d-4225-84f2-6f7e2e445407\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"
Jul 02 14:25:34.250202 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.250124    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/131a76e2-7e0d-4225-84f2-6f7e2e445407-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-76c48f974d-4rrrn\" (UID: \"131a76e2-7e0d-4225-84f2-6f7e2e445407\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"
Jul 02 14:25:34.250202 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.250184    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/131a76e2-7e0d-4225-84f2-6f7e2e445407-model-cache\") pod \"lora-multiple-adapters-test-kserve-76c48f974d-4rrrn\" (UID: \"131a76e2-7e0d-4225-84f2-6f7e2e445407\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"
Jul 02 14:25:34.250351 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.250273    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk25m\" (UniqueName: \"kubernetes.io/projected/131a76e2-7e0d-4225-84f2-6f7e2e445407-kube-api-access-nk25m\") pod \"lora-multiple-adapters-test-kserve-76c48f974d-4rrrn\" (UID: \"131a76e2-7e0d-4225-84f2-6f7e2e445407\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"
Jul 02 14:25:34.250351 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.250304    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/131a76e2-7e0d-4225-84f2-6f7e2e445407-home\") pod \"lora-multiple-adapters-test-kserve-76c48f974d-4rrrn\" (UID: \"131a76e2-7e0d-4225-84f2-6f7e2e445407\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"
Jul 02 14:25:34.250351 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.250329    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/131a76e2-7e0d-4225-84f2-6f7e2e445407-dshm\") pod \"lora-multiple-adapters-test-kserve-76c48f974d-4rrrn\" (UID: \"131a76e2-7e0d-4225-84f2-6f7e2e445407\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"
Jul 02 14:25:34.250605 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.250566    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/131a76e2-7e0d-4225-84f2-6f7e2e445407-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-76c48f974d-4rrrn\" (UID: \"131a76e2-7e0d-4225-84f2-6f7e2e445407\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"
Jul 02 14:25:34.250677 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.250660    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/131a76e2-7e0d-4225-84f2-6f7e2e445407-model-cache\") pod \"lora-multiple-adapters-test-kserve-76c48f974d-4rrrn\" (UID: \"131a76e2-7e0d-4225-84f2-6f7e2e445407\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"
Jul 02 14:25:34.250734 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.250696    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/131a76e2-7e0d-4225-84f2-6f7e2e445407-home\") pod \"lora-multiple-adapters-test-kserve-76c48f974d-4rrrn\" (UID: \"131a76e2-7e0d-4225-84f2-6f7e2e445407\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"
Jul 02 14:25:34.250828 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.250810    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/131a76e2-7e0d-4225-84f2-6f7e2e445407-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-76c48f974d-4rrrn\" (UID: \"131a76e2-7e0d-4225-84f2-6f7e2e445407\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"
Jul 02 14:25:34.250914 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.250894    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/131a76e2-7e0d-4225-84f2-6f7e2e445407-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-76c48f974d-4rrrn\" (UID: \"131a76e2-7e0d-4225-84f2-6f7e2e445407\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"
Jul 02 14:25:34.252521 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.252491    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/131a76e2-7e0d-4225-84f2-6f7e2e445407-dshm\") pod \"lora-multiple-adapters-test-kserve-76c48f974d-4rrrn\" (UID: \"131a76e2-7e0d-4225-84f2-6f7e2e445407\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"
Jul 02 14:25:34.252625 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.252582    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/131a76e2-7e0d-4225-84f2-6f7e2e445407-tls-certs\") pod \"lora-multiple-adapters-test-kserve-76c48f974d-4rrrn\" (UID: \"131a76e2-7e0d-4225-84f2-6f7e2e445407\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"
Jul 02 14:25:34.260539 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.260510    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk25m\" (UniqueName: \"kubernetes.io/projected/131a76e2-7e0d-4225-84f2-6f7e2e445407-kube-api-access-nk25m\") pod \"lora-multiple-adapters-test-kserve-76c48f974d-4rrrn\" (UID: \"131a76e2-7e0d-4225-84f2-6f7e2e445407\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"
Jul 02 14:25:34.375511 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.375476    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"
Jul 02 14:25:34.531933 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:34.531905    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"]
Jul 02 14:25:34.534021 ip-10-0-135-143 kubenswrapper[2583]: W0702 14:25:34.533996    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod131a76e2_7e0d_4225_84f2_6f7e2e445407.slice/crio-135d2e642367be4a1789a97b6e1b5cb554417cda7274eb27ff806fd0eaa300dc WatchSource:0}: Error finding container 135d2e642367be4a1789a97b6e1b5cb554417cda7274eb27ff806fd0eaa300dc: Status 404 returned error can't find the container with id 135d2e642367be4a1789a97b6e1b5cb554417cda7274eb27ff806fd0eaa300dc
Jul 02 14:25:35.017677 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:35.017630    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn" event={"ID":"131a76e2-7e0d-4225-84f2-6f7e2e445407","Type":"ContainerStarted","Data":"f2eefb7bd43368ca8e689e52dea7cbfc410ca0c5c7e8859669a891d1b8d9e373"}
Jul 02 14:25:35.017860 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:35.017730    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn" event={"ID":"131a76e2-7e0d-4225-84f2-6f7e2e445407","Type":"ContainerStarted","Data":"135d2e642367be4a1789a97b6e1b5cb554417cda7274eb27ff806fd0eaa300dc"}
Jul 02 14:25:40.042651 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:40.042618    2583 generic.go:356] "Generic (PLEG): container finished" podID="131a76e2-7e0d-4225-84f2-6f7e2e445407" containerID="f2eefb7bd43368ca8e689e52dea7cbfc410ca0c5c7e8859669a891d1b8d9e373" exitCode=0
Jul 02 14:25:40.043197 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:40.042688    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn" event={"ID":"131a76e2-7e0d-4225-84f2-6f7e2e445407","Type":"ContainerDied","Data":"f2eefb7bd43368ca8e689e52dea7cbfc410ca0c5c7e8859669a891d1b8d9e373"}
Jul 02 14:25:41.050974 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:41.050942    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn" event={"ID":"131a76e2-7e0d-4225-84f2-6f7e2e445407","Type":"ContainerStarted","Data":"a56fa420c86aa3928eb6f4474c7952d89ae75c200060595708efa3a075e5003d"}
Jul 02 14:25:41.080707 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:41.080649    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn" podStartSLOduration=2.635057232 podStartE2EDuration="7.080630008s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.445572776s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:25:41.079322987 +0000 UTC m=+4252.307234162" watchObservedRunningTime="2026-07-02 14:25:41.080630008 +0000 UTC m=+4252.308541474"
Jul 02 14:25:41.376115 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:41.376081    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"
Jul 02 14:25:43.182671 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.182629    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82"]
Jul 02 14:25:43.188199 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.188171    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82"
Jul 02 14:25:43.191739 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.191711    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Jul 02 14:25:43.199261 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.199230    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82"]
Jul 02 14:25:43.225792 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.225759    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:25:43.342853 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.342815    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f41e2054-4f2b-4f75-b441-1df267741320-tmp-dir\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8rh82\" (UID: \"f41e2054-4f2b-4f75-b441-1df267741320\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82"
Jul 02 14:25:43.342853 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.342857    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f41e2054-4f2b-4f75-b441-1df267741320-tls-certs\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8rh82\" (UID: \"f41e2054-4f2b-4f75-b441-1df267741320\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82"
Jul 02 14:25:43.343072 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.342912    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f41e2054-4f2b-4f75-b441-1df267741320-dshm\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8rh82\" (UID: \"f41e2054-4f2b-4f75-b441-1df267741320\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82"
Jul 02 14:25:43.343072 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.342954    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f41e2054-4f2b-4f75-b441-1df267741320-model-cache\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8rh82\" (UID: \"f41e2054-4f2b-4f75-b441-1df267741320\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82"
Jul 02 14:25:43.343072 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.342972    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f41e2054-4f2b-4f75-b441-1df267741320-home\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8rh82\" (UID: \"f41e2054-4f2b-4f75-b441-1df267741320\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82"
Jul 02 14:25:43.343072 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.343046    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkdb2\" (UniqueName: \"kubernetes.io/projected/f41e2054-4f2b-4f75-b441-1df267741320-kube-api-access-gkdb2\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8rh82\" (UID: \"f41e2054-4f2b-4f75-b441-1df267741320\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82"
Jul 02 14:25:43.444040 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.443948    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkdb2\" (UniqueName: \"kubernetes.io/projected/f41e2054-4f2b-4f75-b441-1df267741320-kube-api-access-gkdb2\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8rh82\" (UID: \"f41e2054-4f2b-4f75-b441-1df267741320\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82"
Jul 02 14:25:43.444040 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.444001    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f41e2054-4f2b-4f75-b441-1df267741320-tmp-dir\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8rh82\" (UID: \"f41e2054-4f2b-4f75-b441-1df267741320\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82"
Jul 02 14:25:43.444040 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.444027    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f41e2054-4f2b-4f75-b441-1df267741320-tls-certs\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8rh82\" (UID: \"f41e2054-4f2b-4f75-b441-1df267741320\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82"
Jul 02 14:25:43.444341 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.444063    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f41e2054-4f2b-4f75-b441-1df267741320-dshm\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8rh82\" (UID: \"f41e2054-4f2b-4f75-b441-1df267741320\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82"
Jul 02 14:25:43.444341 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.444109    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f41e2054-4f2b-4f75-b441-1df267741320-model-cache\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8rh82\" (UID: \"f41e2054-4f2b-4f75-b441-1df267741320\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82"
Jul 02 14:25:43.444341 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.444138    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f41e2054-4f2b-4f75-b441-1df267741320-home\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8rh82\" (UID: \"f41e2054-4f2b-4f75-b441-1df267741320\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82"
Jul 02 14:25:43.444535 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.444476    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f41e2054-4f2b-4f75-b441-1df267741320-tmp-dir\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8rh82\" (UID: \"f41e2054-4f2b-4f75-b441-1df267741320\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82"
Jul 02 14:25:43.444535 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.444496    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f41e2054-4f2b-4f75-b441-1df267741320-home\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8rh82\" (UID: \"f41e2054-4f2b-4f75-b441-1df267741320\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82"
Jul 02 14:25:43.444630 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.444566    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f41e2054-4f2b-4f75-b441-1df267741320-model-cache\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8rh82\" (UID: \"f41e2054-4f2b-4f75-b441-1df267741320\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82"
Jul 02 14:25:43.446553 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.446530    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f41e2054-4f2b-4f75-b441-1df267741320-dshm\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8rh82\" (UID: \"f41e2054-4f2b-4f75-b441-1df267741320\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82"
Jul 02 14:25:43.446868 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.446843    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f41e2054-4f2b-4f75-b441-1df267741320-tls-certs\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8rh82\" (UID: \"f41e2054-4f2b-4f75-b441-1df267741320\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82"
Jul 02 14:25:43.453518 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.453488    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkdb2\" (UniqueName: \"kubernetes.io/projected/f41e2054-4f2b-4f75-b441-1df267741320-kube-api-access-gkdb2\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-8rh82\" (UID: \"f41e2054-4f2b-4f75-b441-1df267741320\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82"
Jul 02 14:25:43.502171 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.502127    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82"
Jul 02 14:25:43.517440 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.517380    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p"]
Jul 02 14:25:43.523046 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.522245    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p"
Jul 02 14:25:43.525265 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.525240    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-custom-template-test-epp-sa-dockercfg-bthdb\""
Jul 02 14:25:43.541660 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.541628    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p"]
Jul 02 14:25:43.559603 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.559573    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:25:43.646297 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.646255    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b31-b154-4e00-8c9b-e995a0e7cd36-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567hcd6p\" (UID: \"2e8b1b31-b154-4e00-8c9b-e995a0e7cd36\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p"
Jul 02 14:25:43.646484 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.646368    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b31-b154-4e00-8c9b-e995a0e7cd36-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567hcd6p\" (UID: \"2e8b1b31-b154-4e00-8c9b-e995a0e7cd36\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p"
Jul 02 14:25:43.646484 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.646470    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e8b1b31-b154-4e00-8c9b-e995a0e7cd36-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567hcd6p\" (UID: \"2e8b1b31-b154-4e00-8c9b-e995a0e7cd36\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p"
Jul 02 14:25:43.646567 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.646509    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl578\" (UniqueName: \"kubernetes.io/projected/2e8b1b31-b154-4e00-8c9b-e995a0e7cd36-kube-api-access-zl578\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567hcd6p\" (UID: \"2e8b1b31-b154-4e00-8c9b-e995a0e7cd36\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p"
Jul 02 14:25:43.646662 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.646637    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b31-b154-4e00-8c9b-e995a0e7cd36-tokenizer-uds\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567hcd6p\" (UID: \"2e8b1b31-b154-4e00-8c9b-e995a0e7cd36\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p"
Jul 02 14:25:43.667298 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.667261    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82"]
Jul 02 14:25:43.670959 ip-10-0-135-143 kubenswrapper[2583]: W0702 14:25:43.670883    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf41e2054_4f2b_4f75_b441_1df267741320.slice/crio-c5c63319ac78700ce100ccfba596ecf542b89f8fc3bd6c907d90c36605f93a8f WatchSource:0}: Error finding container c5c63319ac78700ce100ccfba596ecf542b89f8fc3bd6c907d90c36605f93a8f: Status 404 returned error can't find the container with id c5c63319ac78700ce100ccfba596ecf542b89f8fc3bd6c907d90c36605f93a8f
Jul 02 14:25:43.747762 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.747723    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b31-b154-4e00-8c9b-e995a0e7cd36-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567hcd6p\" (UID: \"2e8b1b31-b154-4e00-8c9b-e995a0e7cd36\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p"
Jul 02 14:25:43.747973 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.747816    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e8b1b31-b154-4e00-8c9b-e995a0e7cd36-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567hcd6p\" (UID: \"2e8b1b31-b154-4e00-8c9b-e995a0e7cd36\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p"
Jul 02 14:25:43.747973 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.747851    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl578\" (UniqueName: \"kubernetes.io/projected/2e8b1b31-b154-4e00-8c9b-e995a0e7cd36-kube-api-access-zl578\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567hcd6p\" (UID: \"2e8b1b31-b154-4e00-8c9b-e995a0e7cd36\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p"
Jul 02 14:25:43.747973 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.747961    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b31-b154-4e00-8c9b-e995a0e7cd36-tokenizer-uds\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567hcd6p\" (UID: \"2e8b1b31-b154-4e00-8c9b-e995a0e7cd36\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p"
Jul 02 14:25:43.748148 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.748014    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b31-b154-4e00-8c9b-e995a0e7cd36-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567hcd6p\" (UID: \"2e8b1b31-b154-4e00-8c9b-e995a0e7cd36\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p"
Jul 02 14:25:43.748224 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.748195    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b31-b154-4e00-8c9b-e995a0e7cd36-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567hcd6p\" (UID: \"2e8b1b31-b154-4e00-8c9b-e995a0e7cd36\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p"
Jul 02 14:25:43.748320 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.748291    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b31-b154-4e00-8c9b-e995a0e7cd36-tokenizer-uds\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567hcd6p\" (UID: \"2e8b1b31-b154-4e00-8c9b-e995a0e7cd36\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p"
Jul 02 14:25:43.748428 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.748407    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b31-b154-4e00-8c9b-e995a0e7cd36-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567hcd6p\" (UID: \"2e8b1b31-b154-4e00-8c9b-e995a0e7cd36\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p"
Jul 02 14:25:43.750575 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.750547    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e8b1b31-b154-4e00-8c9b-e995a0e7cd36-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567hcd6p\" (UID: \"2e8b1b31-b154-4e00-8c9b-e995a0e7cd36\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p"
Jul 02 14:25:43.758092 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.758056    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl578\" (UniqueName: \"kubernetes.io/projected/2e8b1b31-b154-4e00-8c9b-e995a0e7cd36-kube-api-access-zl578\") pod \"scheduler-custom-template-test-kserve-router-scheduler-567hcd6p\" (UID: \"2e8b1b31-b154-4e00-8c9b-e995a0e7cd36\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p"
Jul 02 14:25:43.843929 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.843881    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p"
Jul 02 14:25:43.999286 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:43.999096    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p"]
Jul 02 14:25:44.002006 ip-10-0-135-143 kubenswrapper[2583]: W0702 14:25:44.001976    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e8b1b31_b154_4e00_8c9b_e995a0e7cd36.slice/crio-5a6b34163db837bd9fce80d7e8874964d0262e7288fa372b9c2a00962ae27ab3 WatchSource:0}: Error finding container 5a6b34163db837bd9fce80d7e8874964d0262e7288fa372b9c2a00962ae27ab3: Status 404 returned error can't find the container with id 5a6b34163db837bd9fce80d7e8874964d0262e7288fa372b9c2a00962ae27ab3
Jul 02 14:25:44.069160 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:44.069124    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p" event={"ID":"2e8b1b31-b154-4e00-8c9b-e995a0e7cd36","Type":"ContainerStarted","Data":"5a6b34163db837bd9fce80d7e8874964d0262e7288fa372b9c2a00962ae27ab3"}
Jul 02 14:25:44.070715 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:44.070690    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82" event={"ID":"f41e2054-4f2b-4f75-b441-1df267741320","Type":"ContainerStarted","Data":"b7183d952499436a9058751e28a5d23bd88762b468b4cbb492ebb5f64b08722d"}
Jul 02 14:25:44.070863 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:44.070731    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82" event={"ID":"f41e2054-4f2b-4f75-b441-1df267741320","Type":"ContainerStarted","Data":"c5c63319ac78700ce100ccfba596ecf542b89f8fc3bd6c907d90c36605f93a8f"}
Jul 02 14:25:44.091585 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:44.091536    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82" podStartSLOduration=1.091519026 podStartE2EDuration="1.091519026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:25:44.089539268 +0000 UTC m=+4255.317450426" watchObservedRunningTime="2026-07-02 14:25:44.091519026 +0000 UTC m=+4255.319430199"
Jul 02 14:25:44.376176 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:44.376136    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"
Jul 02 14:25:44.377565 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:44.377535    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn" podUID="131a76e2-7e0d-4225-84f2-6f7e2e445407" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 02 14:25:44.502555 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:44.502516    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82"
Jul 02 14:25:45.081994 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:45.081956    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p" event={"ID":"2e8b1b31-b154-4e00-8c9b-e995a0e7cd36","Type":"ContainerStarted","Data":"1e3aae75237eb75f837b1fd1b59c27af7c699f531afdc466c5e60d7918741a07"}
Jul 02 14:25:45.082216 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:45.082016    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p"
Jul 02 14:25:45.100777 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:45.100715    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p" podStartSLOduration=2.100691817 podStartE2EDuration="2.100691817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:25:45.099584464 +0000 UTC m=+4256.327495654" watchObservedRunningTime="2026-07-02 14:25:45.100691817 +0000 UTC m=+4256.328602996"
Jul 02 14:25:53.503279 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:53.503238    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82"
Jul 02 14:25:53.515769 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:53.515733    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82"
Jul 02 14:25:54.142293 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:54.142265    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82"
Jul 02 14:25:54.376077 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:25:54.376035    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn" podUID="131a76e2-7e0d-4225-84f2-6f7e2e445407" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 02 14:26:04.376068 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:26:04.376015    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn" podUID="131a76e2-7e0d-4225-84f2-6f7e2e445407" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 02 14:26:14.376873 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:26:14.376822    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn" podUID="131a76e2-7e0d-4225-84f2-6f7e2e445407" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 02 14:26:15.845149 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:26:15.845113    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:26:16.090225 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:26:16.090194    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p"
Jul 02 14:26:24.376373 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:26:24.376322    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn" podUID="131a76e2-7e0d-4225-84f2-6f7e2e445407" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 02 14:26:34.376696 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:26:34.376631    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn" podUID="131a76e2-7e0d-4225-84f2-6f7e2e445407" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 02 14:26:44.376459 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:26:44.376347    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn" podUID="131a76e2-7e0d-4225-84f2-6f7e2e445407" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 02 14:26:54.376635 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:26:54.376573    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn" podUID="131a76e2-7e0d-4225-84f2-6f7e2e445407" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 02 14:27:04.386217 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:04.386182    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"
Jul 02 14:27:04.394407 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:04.394361    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"
Jul 02 14:27:08.460995 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:08.460959    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p"]
Jul 02 14:27:08.466010 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:08.465974    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82"]
Jul 02 14:27:08.471306 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:27:08.471266    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 02 14:27:08.471306 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 14:27:08.471306 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 14:27:08.471306 ip-10-0-135-143 kubenswrapper[2583]:  > containerID="1e3aae75237eb75f837b1fd1b59c27af7c699f531afdc466c5e60d7918741a07" cmd=["/bin/sleep","15"]
Jul 02 14:27:08.471576 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:08.471327    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p" message="" err=<
Jul 02 14:27:08.471576 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 14:27:08.471576 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 14:27:08.471576 ip-10-0-135-143 kubenswrapper[2583]:  >
Jul 02 14:27:08.471576 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:27:08.471344    2583 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 02 14:27:08.471576 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 14:27:08.471576 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 14:27:08.471576 ip-10-0-135-143 kubenswrapper[2583]:  > pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p" podUID="2e8b1b31-b154-4e00-8c9b-e995a0e7cd36" containerName="main" containerID="cri-o://1e3aae75237eb75f837b1fd1b59c27af7c699f531afdc466c5e60d7918741a07"
Jul 02 14:27:08.471576 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:08.471406    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p" podUID="2e8b1b31-b154-4e00-8c9b-e995a0e7cd36" containerName="main" containerID="cri-o://1e3aae75237eb75f837b1fd1b59c27af7c699f531afdc466c5e60d7918741a07" gracePeriod=60
Jul 02 14:27:08.476375 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:08.476338    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:27:08.532907 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:08.532873    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:27:08.726526 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:08.726460    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p"
Jul 02 14:27:08.807663 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:08.807628    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b31-b154-4e00-8c9b-e995a0e7cd36-cachi2-cache\") pod \"2e8b1b31-b154-4e00-8c9b-e995a0e7cd36\" (UID: \"2e8b1b31-b154-4e00-8c9b-e995a0e7cd36\") "
Jul 02 14:27:08.807848 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:08.807675    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zl578\" (UniqueName: \"kubernetes.io/projected/2e8b1b31-b154-4e00-8c9b-e995a0e7cd36-kube-api-access-zl578\") pod \"2e8b1b31-b154-4e00-8c9b-e995a0e7cd36\" (UID: \"2e8b1b31-b154-4e00-8c9b-e995a0e7cd36\") "
Jul 02 14:27:08.807848 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:08.807713    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b31-b154-4e00-8c9b-e995a0e7cd36-tokenizer-cache\") pod \"2e8b1b31-b154-4e00-8c9b-e995a0e7cd36\" (UID: \"2e8b1b31-b154-4e00-8c9b-e995a0e7cd36\") "
Jul 02 14:27:08.807848 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:08.807768    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b31-b154-4e00-8c9b-e995a0e7cd36-tokenizer-uds\") pod \"2e8b1b31-b154-4e00-8c9b-e995a0e7cd36\" (UID: \"2e8b1b31-b154-4e00-8c9b-e995a0e7cd36\") "
Jul 02 14:27:08.807848 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:08.807814    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e8b1b31-b154-4e00-8c9b-e995a0e7cd36-tls-certs\") pod \"2e8b1b31-b154-4e00-8c9b-e995a0e7cd36\" (UID: \"2e8b1b31-b154-4e00-8c9b-e995a0e7cd36\") "
Jul 02 14:27:08.808020 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:08.807881    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8b1b31-b154-4e00-8c9b-e995a0e7cd36-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "2e8b1b31-b154-4e00-8c9b-e995a0e7cd36" (UID: "2e8b1b31-b154-4e00-8c9b-e995a0e7cd36"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:27:08.808020 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:08.807987    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8b1b31-b154-4e00-8c9b-e995a0e7cd36-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "2e8b1b31-b154-4e00-8c9b-e995a0e7cd36" (UID: "2e8b1b31-b154-4e00-8c9b-e995a0e7cd36"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:27:08.808099 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:08.808055    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8b1b31-b154-4e00-8c9b-e995a0e7cd36-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "2e8b1b31-b154-4e00-8c9b-e995a0e7cd36" (UID: "2e8b1b31-b154-4e00-8c9b-e995a0e7cd36"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:27:08.808136 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:08.808129    2583 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b31-b154-4e00-8c9b-e995a0e7cd36-cachi2-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:27:08.808195 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:08.808142    2583 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b31-b154-4e00-8c9b-e995a0e7cd36-tokenizer-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:27:08.808195 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:08.808153    2583 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b31-b154-4e00-8c9b-e995a0e7cd36-tokenizer-uds\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:27:08.809898 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:08.809875    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e8b1b31-b154-4e00-8c9b-e995a0e7cd36-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e8b1b31-b154-4e00-8c9b-e995a0e7cd36" (UID: "2e8b1b31-b154-4e00-8c9b-e995a0e7cd36"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:27:08.809983 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:08.809918    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e8b1b31-b154-4e00-8c9b-e995a0e7cd36-kube-api-access-zl578" (OuterVolumeSpecName: "kube-api-access-zl578") pod "2e8b1b31-b154-4e00-8c9b-e995a0e7cd36" (UID: "2e8b1b31-b154-4e00-8c9b-e995a0e7cd36"). InnerVolumeSpecName "kube-api-access-zl578". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:27:08.909126 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:08.909073    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e8b1b31-b154-4e00-8c9b-e995a0e7cd36-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:27:08.909126 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:08.909119    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl578\" (UniqueName: \"kubernetes.io/projected/2e8b1b31-b154-4e00-8c9b-e995a0e7cd36-kube-api-access-zl578\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:27:09.542072 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:09.541986    2583 generic.go:356] "Generic (PLEG): container finished" podID="2e8b1b31-b154-4e00-8c9b-e995a0e7cd36" containerID="1e3aae75237eb75f837b1fd1b59c27af7c699f531afdc466c5e60d7918741a07" exitCode=0
Jul 02 14:27:09.542072 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:09.542035    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p"
Jul 02 14:27:09.542563 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:09.542073    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p" event={"ID":"2e8b1b31-b154-4e00-8c9b-e995a0e7cd36","Type":"ContainerDied","Data":"1e3aae75237eb75f837b1fd1b59c27af7c699f531afdc466c5e60d7918741a07"}
Jul 02 14:27:09.542563 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:09.542123    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p" event={"ID":"2e8b1b31-b154-4e00-8c9b-e995a0e7cd36","Type":"ContainerDied","Data":"5a6b34163db837bd9fce80d7e8874964d0262e7288fa372b9c2a00962ae27ab3"}
Jul 02 14:27:09.542563 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:09.542145    2583 scope.go:117] "RemoveContainer" containerID="1e3aae75237eb75f837b1fd1b59c27af7c699f531afdc466c5e60d7918741a07"
Jul 02 14:27:09.551535 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:09.551513    2583 scope.go:117] "RemoveContainer" containerID="1e3aae75237eb75f837b1fd1b59c27af7c699f531afdc466c5e60d7918741a07"
Jul 02 14:27:09.551797 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:27:09.551779    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e3aae75237eb75f837b1fd1b59c27af7c699f531afdc466c5e60d7918741a07\": container with ID starting with 1e3aae75237eb75f837b1fd1b59c27af7c699f531afdc466c5e60d7918741a07 not found: ID does not exist" containerID="1e3aae75237eb75f837b1fd1b59c27af7c699f531afdc466c5e60d7918741a07"
Jul 02 14:27:09.551844 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:09.551804    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e3aae75237eb75f837b1fd1b59c27af7c699f531afdc466c5e60d7918741a07"} err="failed to get container status \"1e3aae75237eb75f837b1fd1b59c27af7c699f531afdc466c5e60d7918741a07\": rpc error: code = NotFound desc = could not find container \"1e3aae75237eb75f837b1fd1b59c27af7c699f531afdc466c5e60d7918741a07\": container with ID starting with 1e3aae75237eb75f837b1fd1b59c27af7c699f531afdc466c5e60d7918741a07 not found: ID does not exist"
Jul 02 14:27:09.560685 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:09.560658    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p"]
Jul 02 14:27:09.567039 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:09.567011    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-567hcd6p"]
Jul 02 14:27:11.233866 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:11.233829    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e8b1b31-b154-4e00-8c9b-e995a0e7cd36" path="/var/lib/kubelet/pods/2e8b1b31-b154-4e00-8c9b-e995a0e7cd36/volumes"
Jul 02 14:27:14.082978 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:14.082941    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:27:23.490056 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:23.489975    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82" podUID="f41e2054-4f2b-4f75-b441-1df267741320" containerName="main" containerID="cri-o://b7183d952499436a9058751e28a5d23bd88762b468b4cbb492ebb5f64b08722d" gracePeriod=45
Jul 02 14:27:23.616429 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:23.616375    2583 generic.go:356] "Generic (PLEG): container finished" podID="f41e2054-4f2b-4f75-b441-1df267741320" containerID="b7183d952499436a9058751e28a5d23bd88762b468b4cbb492ebb5f64b08722d" exitCode=0
Jul 02 14:27:23.616727 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:23.616489    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82" event={"ID":"f41e2054-4f2b-4f75-b441-1df267741320","Type":"ContainerDied","Data":"b7183d952499436a9058751e28a5d23bd88762b468b4cbb492ebb5f64b08722d"}
Jul 02 14:27:23.748937 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:23.748858    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82"
Jul 02 14:27:23.860314 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:23.860261    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkdb2\" (UniqueName: \"kubernetes.io/projected/f41e2054-4f2b-4f75-b441-1df267741320-kube-api-access-gkdb2\") pod \"f41e2054-4f2b-4f75-b441-1df267741320\" (UID: \"f41e2054-4f2b-4f75-b441-1df267741320\") "
Jul 02 14:27:23.860498 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:23.860422    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f41e2054-4f2b-4f75-b441-1df267741320-model-cache\") pod \"f41e2054-4f2b-4f75-b441-1df267741320\" (UID: \"f41e2054-4f2b-4f75-b441-1df267741320\") "
Jul 02 14:27:23.860498 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:23.860473    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f41e2054-4f2b-4f75-b441-1df267741320-home\") pod \"f41e2054-4f2b-4f75-b441-1df267741320\" (UID: \"f41e2054-4f2b-4f75-b441-1df267741320\") "
Jul 02 14:27:23.860618 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:23.860508    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f41e2054-4f2b-4f75-b441-1df267741320-tls-certs\") pod \"f41e2054-4f2b-4f75-b441-1df267741320\" (UID: \"f41e2054-4f2b-4f75-b441-1df267741320\") "
Jul 02 14:27:23.860925 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:23.860899    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f41e2054-4f2b-4f75-b441-1df267741320-dshm\") pod \"f41e2054-4f2b-4f75-b441-1df267741320\" (UID: \"f41e2054-4f2b-4f75-b441-1df267741320\") "
Jul 02 14:27:23.861055 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:23.860949    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f41e2054-4f2b-4f75-b441-1df267741320-tmp-dir\") pod \"f41e2054-4f2b-4f75-b441-1df267741320\" (UID: \"f41e2054-4f2b-4f75-b441-1df267741320\") "
Jul 02 14:27:23.861315 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:23.860752    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f41e2054-4f2b-4f75-b441-1df267741320-home" (OuterVolumeSpecName: "home") pod "f41e2054-4f2b-4f75-b441-1df267741320" (UID: "f41e2054-4f2b-4f75-b441-1df267741320"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:27:23.861433 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:23.860775    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f41e2054-4f2b-4f75-b441-1df267741320-model-cache" (OuterVolumeSpecName: "model-cache") pod "f41e2054-4f2b-4f75-b441-1df267741320" (UID: "f41e2054-4f2b-4f75-b441-1df267741320"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:27:23.861573 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:23.861544    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f41e2054-4f2b-4f75-b441-1df267741320-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f41e2054-4f2b-4f75-b441-1df267741320" (UID: "f41e2054-4f2b-4f75-b441-1df267741320"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:27:23.863085 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:23.863054    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f41e2054-4f2b-4f75-b441-1df267741320-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f41e2054-4f2b-4f75-b441-1df267741320" (UID: "f41e2054-4f2b-4f75-b441-1df267741320"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:27:23.863171 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:23.863076    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f41e2054-4f2b-4f75-b441-1df267741320-kube-api-access-gkdb2" (OuterVolumeSpecName: "kube-api-access-gkdb2") pod "f41e2054-4f2b-4f75-b441-1df267741320" (UID: "f41e2054-4f2b-4f75-b441-1df267741320"). InnerVolumeSpecName "kube-api-access-gkdb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:27:23.863654 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:23.863634    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f41e2054-4f2b-4f75-b441-1df267741320-dshm" (OuterVolumeSpecName: "dshm") pod "f41e2054-4f2b-4f75-b441-1df267741320" (UID: "f41e2054-4f2b-4f75-b441-1df267741320"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:27:23.962116 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:23.962072    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f41e2054-4f2b-4f75-b441-1df267741320-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:27:23.962116 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:23.962104    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f41e2054-4f2b-4f75-b441-1df267741320-home\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:27:23.962116 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:23.962113    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f41e2054-4f2b-4f75-b441-1df267741320-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:27:23.962116 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:23.962122    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f41e2054-4f2b-4f75-b441-1df267741320-dshm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:27:23.962116 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:23.962129    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f41e2054-4f2b-4f75-b441-1df267741320-tmp-dir\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:27:23.962535 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:23.962139    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkdb2\" (UniqueName: \"kubernetes.io/projected/f41e2054-4f2b-4f75-b441-1df267741320-kube-api-access-gkdb2\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:27:24.623249 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:24.623219    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82"
Jul 02 14:27:24.623703 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:24.623243    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82" event={"ID":"f41e2054-4f2b-4f75-b441-1df267741320","Type":"ContainerDied","Data":"c5c63319ac78700ce100ccfba596ecf542b89f8fc3bd6c907d90c36605f93a8f"}
Jul 02 14:27:24.623703 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:24.623312    2583 scope.go:117] "RemoveContainer" containerID="b7183d952499436a9058751e28a5d23bd88762b468b4cbb492ebb5f64b08722d"
Jul 02 14:27:24.647929 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:24.647890    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82"]
Jul 02 14:27:24.651359 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:24.651332    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-8rh82"]
Jul 02 14:27:25.235160 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:25.235125    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f41e2054-4f2b-4f75-b441-1df267741320" path="/var/lib/kubelet/pods/f41e2054-4f2b-4f75-b441-1df267741320/volumes"
Jul 02 14:27:50.166823 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.166784    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"]
Jul 02 14:27:50.167253 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.167204    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e8b1b31-b154-4e00-8c9b-e995a0e7cd36" containerName="main"
Jul 02 14:27:50.167253 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.167217    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e8b1b31-b154-4e00-8c9b-e995a0e7cd36" containerName="main"
Jul 02 14:27:50.167253 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.167243    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f41e2054-4f2b-4f75-b441-1df267741320" containerName="main"
Jul 02 14:27:50.167253 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.167250    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f41e2054-4f2b-4f75-b441-1df267741320" containerName="main"
Jul 02 14:27:50.167441 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.167321    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f41e2054-4f2b-4f75-b441-1df267741320" containerName="main"
Jul 02 14:27:50.167441 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.167333    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e8b1b31-b154-4e00-8c9b-e995a0e7cd36" containerName="main"
Jul 02 14:27:50.173382 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.173353    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"
Jul 02 14:27:50.176119 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.176095    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-nzh24\""
Jul 02 14:27:50.176276 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.176128    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 02 14:27:50.184081 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.184052    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"]
Jul 02 14:27:50.234788 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.234682    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vc7t\" (UniqueName: \"kubernetes.io/projected/89360742-358c-4308-b211-e677a5fa700f-kube-api-access-9vc7t\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl\" (UID: \"89360742-358c-4308-b211-e677a5fa700f\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"
Jul 02 14:27:50.234788 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.234727    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89360742-358c-4308-b211-e677a5fa700f-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl\" (UID: \"89360742-358c-4308-b211-e677a5fa700f\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"
Jul 02 14:27:50.235011 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.234793    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89360742-358c-4308-b211-e677a5fa700f-home\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl\" (UID: \"89360742-358c-4308-b211-e677a5fa700f\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"
Jul 02 14:27:50.235011 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.234826    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89360742-358c-4308-b211-e677a5fa700f-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl\" (UID: \"89360742-358c-4308-b211-e677a5fa700f\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"
Jul 02 14:27:50.235011 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.234857    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89360742-358c-4308-b211-e677a5fa700f-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl\" (UID: \"89360742-358c-4308-b211-e677a5fa700f\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"
Jul 02 14:27:50.235011 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.234904    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89360742-358c-4308-b211-e677a5fa700f-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl\" (UID: \"89360742-358c-4308-b211-e677a5fa700f\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"
Jul 02 14:27:50.235734 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.235699    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:27:50.336048 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.335982    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89360742-358c-4308-b211-e677a5fa700f-home\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl\" (UID: \"89360742-358c-4308-b211-e677a5fa700f\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"
Jul 02 14:27:50.336279 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.336051    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89360742-358c-4308-b211-e677a5fa700f-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl\" (UID: \"89360742-358c-4308-b211-e677a5fa700f\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"
Jul 02 14:27:50.336279 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.336092    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89360742-358c-4308-b211-e677a5fa700f-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl\" (UID: \"89360742-358c-4308-b211-e677a5fa700f\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"
Jul 02 14:27:50.336279 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.336162    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89360742-358c-4308-b211-e677a5fa700f-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl\" (UID: \"89360742-358c-4308-b211-e677a5fa700f\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"
Jul 02 14:27:50.336495 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.336309    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vc7t\" (UniqueName: \"kubernetes.io/projected/89360742-358c-4308-b211-e677a5fa700f-kube-api-access-9vc7t\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl\" (UID: \"89360742-358c-4308-b211-e677a5fa700f\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"
Jul 02 14:27:50.336495 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.336352    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89360742-358c-4308-b211-e677a5fa700f-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl\" (UID: \"89360742-358c-4308-b211-e677a5fa700f\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"
Jul 02 14:27:50.337289 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.337262    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89360742-358c-4308-b211-e677a5fa700f-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl\" (UID: \"89360742-358c-4308-b211-e677a5fa700f\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"
Jul 02 14:27:50.337766 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.337700    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89360742-358c-4308-b211-e677a5fa700f-home\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl\" (UID: \"89360742-358c-4308-b211-e677a5fa700f\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"
Jul 02 14:27:50.338157 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.338126    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89360742-358c-4308-b211-e677a5fa700f-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl\" (UID: \"89360742-358c-4308-b211-e677a5fa700f\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"
Jul 02 14:27:50.345859 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.345790    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89360742-358c-4308-b211-e677a5fa700f-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl\" (UID: \"89360742-358c-4308-b211-e677a5fa700f\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"
Jul 02 14:27:50.347085 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.347024    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89360742-358c-4308-b211-e677a5fa700f-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl\" (UID: \"89360742-358c-4308-b211-e677a5fa700f\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"
Jul 02 14:27:50.350519 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.349188    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vc7t\" (UniqueName: \"kubernetes.io/projected/89360742-358c-4308-b211-e677a5fa700f-kube-api-access-9vc7t\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl\" (UID: \"89360742-358c-4308-b211-e677a5fa700f\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"
Jul 02 14:27:50.406307 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.406276    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj"]
Jul 02 14:27:50.411153 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.411126    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj"
Jul 02 14:27:50.415046 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.415017    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-fwwxr\""
Jul 02 14:27:50.421209 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.421138    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj"]
Jul 02 14:27:50.446328 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.446251    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:27:50.487671 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.487571    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"
Jul 02 14:27:50.543500 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.543427    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj\" (UID: \"5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj"
Jul 02 14:27:50.543718 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.543586    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj\" (UID: \"5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj"
Jul 02 14:27:50.543718 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.543635    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42j4n\" (UniqueName: \"kubernetes.io/projected/5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52-kube-api-access-42j4n\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj\" (UID: \"5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj"
Jul 02 14:27:50.645262 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.645206    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj\" (UID: \"5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj"
Jul 02 14:27:50.645595 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.645284    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42j4n\" (UniqueName: \"kubernetes.io/projected/5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52-kube-api-access-42j4n\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj\" (UID: \"5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj"
Jul 02 14:27:50.645595 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.645431    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj\" (UID: \"5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj"
Jul 02 14:27:50.646028 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.645979    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj\" (UID: \"5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj"
Jul 02 14:27:50.650132 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.650094    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj\" (UID: \"5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj"
Jul 02 14:27:50.666007 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.665982    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42j4n\" (UniqueName: \"kubernetes.io/projected/5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52-kube-api-access-42j4n\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj\" (UID: \"5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj"
Jul 02 14:27:50.666971 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.666950    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"]
Jul 02 14:27:50.668772 ip-10-0-135-143 kubenswrapper[2583]: W0702 14:27:50.668750    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89360742_358c_4308_b211_e677a5fa700f.slice/crio-af11b37bfd08d9927af8cf6881e57184abfab61ba815cb8f67227934babd7948 WatchSource:0}: Error finding container af11b37bfd08d9927af8cf6881e57184abfab61ba815cb8f67227934babd7948: Status 404 returned error can't find the container with id af11b37bfd08d9927af8cf6881e57184abfab61ba815cb8f67227934babd7948
Jul 02 14:27:50.670668 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.670652    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 14:27:50.730673 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.730622    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj"
Jul 02 14:27:50.766383 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.766347    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl" event={"ID":"89360742-358c-4308-b211-e677a5fa700f","Type":"ContainerStarted","Data":"af11b37bfd08d9927af8cf6881e57184abfab61ba815cb8f67227934babd7948"}
Jul 02 14:27:50.904495 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:50.904442    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj"]
Jul 02 14:27:50.907311 ip-10-0-135-143 kubenswrapper[2583]: W0702 14:27:50.907265    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c8cd6b4_0901_49c6_9eb7_b2c9e6439c52.slice/crio-fe143ec7f938e6f2853ba6fc59d80f6f897011f3a95f4531828d4662252d54e2 WatchSource:0}: Error finding container fe143ec7f938e6f2853ba6fc59d80f6f897011f3a95f4531828d4662252d54e2: Status 404 returned error can't find the container with id fe143ec7f938e6f2853ba6fc59d80f6f897011f3a95f4531828d4662252d54e2
Jul 02 14:27:51.776886 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:51.776851    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj" event={"ID":"5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52","Type":"ContainerStarted","Data":"9fb36a9be1fea939f88617a85ea1a76432d1f6fdcc733ba133ec195729fafc24"}
Jul 02 14:27:51.777277 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:51.776912    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj" event={"ID":"5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52","Type":"ContainerStarted","Data":"fe143ec7f938e6f2853ba6fc59d80f6f897011f3a95f4531828d4662252d54e2"}
Jul 02 14:27:51.777277 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:51.776938    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj"
Jul 02 14:27:51.798010 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:51.797949    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj" podStartSLOduration=1.797932208 podStartE2EDuration="1.797932208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:27:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:27:51.795858292 +0000 UTC m=+4383.023769467" watchObservedRunningTime="2026-07-02 14:27:51.797932208 +0000 UTC m=+4383.025843381"
Jul 02 14:27:52.783784 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:52.783744    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl" event={"ID":"89360742-358c-4308-b211-e677a5fa700f","Type":"ContainerStarted","Data":"1f792493590971ed5a224762b3f1e9be85d3c67b98b7f1a7573c9d060a1d0d61"}
Jul 02 14:27:52.784193 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:52.783823    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"
Jul 02 14:27:53.790242 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:53.790208    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl" event={"ID":"89360742-358c-4308-b211-e677a5fa700f","Type":"ContainerStarted","Data":"cc7cb59c8f681f6305c26f684fe51ec5d15f0a236a418f1e42fa6b4fcf35c3ee"}
Jul 02 14:27:53.814481 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:53.814365    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl" podStartSLOduration=2.786228199 podStartE2EDuration="3.814337134s" totalImagesPullingTime="1.028108935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:27:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:27:53.81066835 +0000 UTC m=+4385.038579538" watchObservedRunningTime="2026-07-02 14:27:53.814337134 +0000 UTC m=+4385.042248317"
Jul 02 14:27:54.488058 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:27:54.488021    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"
Jul 02 14:28:00.488455 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:00.488417    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"
Jul 02 14:28:00.500846 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:00.500819    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"
Jul 02 14:28:00.852703 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:00.852614    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"
Jul 02 14:28:01.861099 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:01.861072    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"
Jul 02 14:28:15.281171 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:15.281082    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"]
Jul 02 14:28:15.298964 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:15.298922    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:28:22.786117 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:22.786080    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj"
Jul 02 14:28:30.306845 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:30.306782    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn" podUID="131a76e2-7e0d-4225-84f2-6f7e2e445407" containerName="main" containerID="cri-o://a56fa420c86aa3928eb6f4474c7952d89ae75c200060595708efa3a075e5003d" gracePeriod=45
Jul 02 14:28:30.387620 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:30.387574    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn" podUID="131a76e2-7e0d-4225-84f2-6f7e2e445407" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 02 14:28:30.605227 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:30.605132    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:28:30.979522 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:30.979496    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"
Jul 02 14:28:30.996549 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:30.996516    2583 generic.go:356] "Generic (PLEG): container finished" podID="131a76e2-7e0d-4225-84f2-6f7e2e445407" containerID="a56fa420c86aa3928eb6f4474c7952d89ae75c200060595708efa3a075e5003d" exitCode=0
Jul 02 14:28:30.996726 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:30.996554    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn" event={"ID":"131a76e2-7e0d-4225-84f2-6f7e2e445407","Type":"ContainerDied","Data":"a56fa420c86aa3928eb6f4474c7952d89ae75c200060595708efa3a075e5003d"}
Jul 02 14:28:30.996726 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:30.996582    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"
Jul 02 14:28:30.996726 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:30.996629    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn" event={"ID":"131a76e2-7e0d-4225-84f2-6f7e2e445407","Type":"ContainerDied","Data":"135d2e642367be4a1789a97b6e1b5cb554417cda7274eb27ff806fd0eaa300dc"}
Jul 02 14:28:30.996726 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:30.996661    2583 scope.go:117] "RemoveContainer" containerID="a56fa420c86aa3928eb6f4474c7952d89ae75c200060595708efa3a075e5003d"
Jul 02 14:28:31.008036 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:31.008010    2583 scope.go:117] "RemoveContainer" containerID="f2eefb7bd43368ca8e689e52dea7cbfc410ca0c5c7e8859669a891d1b8d9e373"
Jul 02 14:28:31.018705 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:31.018679    2583 scope.go:117] "RemoveContainer" containerID="a56fa420c86aa3928eb6f4474c7952d89ae75c200060595708efa3a075e5003d"
Jul 02 14:28:31.019018 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:28:31.018996    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a56fa420c86aa3928eb6f4474c7952d89ae75c200060595708efa3a075e5003d\": container with ID starting with a56fa420c86aa3928eb6f4474c7952d89ae75c200060595708efa3a075e5003d not found: ID does not exist" containerID="a56fa420c86aa3928eb6f4474c7952d89ae75c200060595708efa3a075e5003d"
Jul 02 14:28:31.019102 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:31.019025    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a56fa420c86aa3928eb6f4474c7952d89ae75c200060595708efa3a075e5003d"} err="failed to get container status \"a56fa420c86aa3928eb6f4474c7952d89ae75c200060595708efa3a075e5003d\": rpc error: code = NotFound desc = could not find container \"a56fa420c86aa3928eb6f4474c7952d89ae75c200060595708efa3a075e5003d\": container with ID starting with a56fa420c86aa3928eb6f4474c7952d89ae75c200060595708efa3a075e5003d not found: ID does not exist"
Jul 02 14:28:31.019102 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:31.019045    2583 scope.go:117] "RemoveContainer" containerID="f2eefb7bd43368ca8e689e52dea7cbfc410ca0c5c7e8859669a891d1b8d9e373"
Jul 02 14:28:31.019330 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:28:31.019305    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2eefb7bd43368ca8e689e52dea7cbfc410ca0c5c7e8859669a891d1b8d9e373\": container with ID starting with f2eefb7bd43368ca8e689e52dea7cbfc410ca0c5c7e8859669a891d1b8d9e373 not found: ID does not exist" containerID="f2eefb7bd43368ca8e689e52dea7cbfc410ca0c5c7e8859669a891d1b8d9e373"
Jul 02 14:28:31.019420 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:31.019335    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2eefb7bd43368ca8e689e52dea7cbfc410ca0c5c7e8859669a891d1b8d9e373"} err="failed to get container status \"f2eefb7bd43368ca8e689e52dea7cbfc410ca0c5c7e8859669a891d1b8d9e373\": rpc error: code = NotFound desc = could not find container \"f2eefb7bd43368ca8e689e52dea7cbfc410ca0c5c7e8859669a891d1b8d9e373\": container with ID starting with f2eefb7bd43368ca8e689e52dea7cbfc410ca0c5c7e8859669a891d1b8d9e373 not found: ID does not exist"
Jul 02 14:28:31.061492 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:31.061453    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/131a76e2-7e0d-4225-84f2-6f7e2e445407-tls-certs\") pod \"131a76e2-7e0d-4225-84f2-6f7e2e445407\" (UID: \"131a76e2-7e0d-4225-84f2-6f7e2e445407\") "
Jul 02 14:28:31.061694 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:31.061537    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/131a76e2-7e0d-4225-84f2-6f7e2e445407-model-cache\") pod \"131a76e2-7e0d-4225-84f2-6f7e2e445407\" (UID: \"131a76e2-7e0d-4225-84f2-6f7e2e445407\") "
Jul 02 14:28:31.061694 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:31.061559    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/131a76e2-7e0d-4225-84f2-6f7e2e445407-dshm\") pod \"131a76e2-7e0d-4225-84f2-6f7e2e445407\" (UID: \"131a76e2-7e0d-4225-84f2-6f7e2e445407\") "
Jul 02 14:28:31.061694 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:31.061586    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/131a76e2-7e0d-4225-84f2-6f7e2e445407-home\") pod \"131a76e2-7e0d-4225-84f2-6f7e2e445407\" (UID: \"131a76e2-7e0d-4225-84f2-6f7e2e445407\") "
Jul 02 14:28:31.061694 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:31.061611    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/131a76e2-7e0d-4225-84f2-6f7e2e445407-kserve-provision-location\") pod \"131a76e2-7e0d-4225-84f2-6f7e2e445407\" (UID: \"131a76e2-7e0d-4225-84f2-6f7e2e445407\") "
Jul 02 14:28:31.061694 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:31.061679    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/131a76e2-7e0d-4225-84f2-6f7e2e445407-cabundle-cert\") pod \"131a76e2-7e0d-4225-84f2-6f7e2e445407\" (UID: \"131a76e2-7e0d-4225-84f2-6f7e2e445407\") "
Jul 02 14:28:31.061960 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:31.061720    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/131a76e2-7e0d-4225-84f2-6f7e2e445407-tmp-dir\") pod \"131a76e2-7e0d-4225-84f2-6f7e2e445407\" (UID: \"131a76e2-7e0d-4225-84f2-6f7e2e445407\") "
Jul 02 14:28:31.061960 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:31.061746    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nk25m\" (UniqueName: \"kubernetes.io/projected/131a76e2-7e0d-4225-84f2-6f7e2e445407-kube-api-access-nk25m\") pod \"131a76e2-7e0d-4225-84f2-6f7e2e445407\" (UID: \"131a76e2-7e0d-4225-84f2-6f7e2e445407\") "
Jul 02 14:28:31.061960 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:31.061865    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131a76e2-7e0d-4225-84f2-6f7e2e445407-model-cache" (OuterVolumeSpecName: "model-cache") pod "131a76e2-7e0d-4225-84f2-6f7e2e445407" (UID: "131a76e2-7e0d-4225-84f2-6f7e2e445407"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:28:31.062094 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:31.061976    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131a76e2-7e0d-4225-84f2-6f7e2e445407-home" (OuterVolumeSpecName: "home") pod "131a76e2-7e0d-4225-84f2-6f7e2e445407" (UID: "131a76e2-7e0d-4225-84f2-6f7e2e445407"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:28:31.062152 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:31.062139    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/131a76e2-7e0d-4225-84f2-6f7e2e445407-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:28:31.062200 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:31.062159    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/131a76e2-7e0d-4225-84f2-6f7e2e445407-home\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:28:31.062264 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:31.062240    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/131a76e2-7e0d-4225-84f2-6f7e2e445407-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "131a76e2-7e0d-4225-84f2-6f7e2e445407" (UID: "131a76e2-7e0d-4225-84f2-6f7e2e445407"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:28:31.063914 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:31.063883    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131a76e2-7e0d-4225-84f2-6f7e2e445407-dshm" (OuterVolumeSpecName: "dshm") pod "131a76e2-7e0d-4225-84f2-6f7e2e445407" (UID: "131a76e2-7e0d-4225-84f2-6f7e2e445407"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:28:31.063914 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:31.063901    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/131a76e2-7e0d-4225-84f2-6f7e2e445407-kube-api-access-nk25m" (OuterVolumeSpecName: "kube-api-access-nk25m") pod "131a76e2-7e0d-4225-84f2-6f7e2e445407" (UID: "131a76e2-7e0d-4225-84f2-6f7e2e445407"). InnerVolumeSpecName "kube-api-access-nk25m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:28:31.064082 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:31.064062    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/131a76e2-7e0d-4225-84f2-6f7e2e445407-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "131a76e2-7e0d-4225-84f2-6f7e2e445407" (UID: "131a76e2-7e0d-4225-84f2-6f7e2e445407"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:28:31.072761 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:31.072726    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131a76e2-7e0d-4225-84f2-6f7e2e445407-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "131a76e2-7e0d-4225-84f2-6f7e2e445407" (UID: "131a76e2-7e0d-4225-84f2-6f7e2e445407"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:28:31.128571 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:31.128518    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131a76e2-7e0d-4225-84f2-6f7e2e445407-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "131a76e2-7e0d-4225-84f2-6f7e2e445407" (UID: "131a76e2-7e0d-4225-84f2-6f7e2e445407"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:28:31.162745 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:31.162657    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/131a76e2-7e0d-4225-84f2-6f7e2e445407-dshm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:28:31.162745 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:31.162686    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/131a76e2-7e0d-4225-84f2-6f7e2e445407-kserve-provision-location\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:28:31.162745 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:31.162696    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/131a76e2-7e0d-4225-84f2-6f7e2e445407-cabundle-cert\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:28:31.162745 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:31.162708    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/131a76e2-7e0d-4225-84f2-6f7e2e445407-tmp-dir\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:28:31.162745 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:31.162716    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk25m\" (UniqueName: \"kubernetes.io/projected/131a76e2-7e0d-4225-84f2-6f7e2e445407-kube-api-access-nk25m\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:28:31.162745 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:31.162724    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/131a76e2-7e0d-4225-84f2-6f7e2e445407-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:28:31.314253 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:31.314216    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"]
Jul 02 14:28:31.317918 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:31.317888    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-76c48f974d-4rrrn"]
Jul 02 14:28:33.234316 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:33.234279    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="131a76e2-7e0d-4225-84f2-6f7e2e445407" path="/var/lib/kubelet/pods/131a76e2-7e0d-4225-84f2-6f7e2e445407/volumes"
Jul 02 14:28:55.403663 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:55.403616    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj"]
Jul 02 14:28:55.406148 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:55.406118    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"]
Jul 02 14:28:55.413801 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:28:55.413764    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 02 14:28:55.413801 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 14:28:55.413801 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 14:28:55.413801 ip-10-0-135-143 kubenswrapper[2583]:  > containerID="9fb36a9be1fea939f88617a85ea1a76432d1f6fdcc733ba133ec195729fafc24" cmd=["/bin/sleep","15"]
Jul 02 14:28:55.414122 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:55.413821    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj" message="" err=<
Jul 02 14:28:55.414122 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 14:28:55.414122 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 14:28:55.414122 ip-10-0-135-143 kubenswrapper[2583]:  >
Jul 02 14:28:55.414122 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:28:55.413837    2583 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 02 14:28:55.414122 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 14:28:55.414122 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 14:28:55.414122 ip-10-0-135-143 kubenswrapper[2583]:  > pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj" podUID="5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52" containerName="main" containerID="cri-o://9fb36a9be1fea939f88617a85ea1a76432d1f6fdcc733ba133ec195729fafc24"
Jul 02 14:28:55.414122 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:55.413874    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj" podUID="5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52" containerName="main" containerID="cri-o://9fb36a9be1fea939f88617a85ea1a76432d1f6fdcc733ba133ec195729fafc24" gracePeriod=60
Jul 02 14:28:55.415029 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:55.414867    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:28:55.515110 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:55.515070    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:28:55.669473 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:55.669445    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj"
Jul 02 14:28:55.791128 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:55.791095    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52-tokenizer-uds\") pod \"5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52\" (UID: \"5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52\") "
Jul 02 14:28:55.791128 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:55.791129    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52-tls-certs\") pod \"5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52\" (UID: \"5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52\") "
Jul 02 14:28:55.791427 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:55.791163    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42j4n\" (UniqueName: \"kubernetes.io/projected/5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52-kube-api-access-42j4n\") pod \"5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52\" (UID: \"5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52\") "
Jul 02 14:28:55.791496 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:55.791421    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52" (UID: "5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:28:55.793322 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:55.793291    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52-kube-api-access-42j4n" (OuterVolumeSpecName: "kube-api-access-42j4n") pod "5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52" (UID: "5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52"). InnerVolumeSpecName "kube-api-access-42j4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:28:55.793460 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:55.793341    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52" (UID: "5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:28:55.892556 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:55.892514    2583 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52-tokenizer-uds\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:28:55.892556 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:55.892547    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:28:55.892556 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:55.892557    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42j4n\" (UniqueName: \"kubernetes.io/projected/5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52-kube-api-access-42j4n\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:28:56.123533 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:56.123496    2583 generic.go:356] "Generic (PLEG): container finished" podID="5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52" containerID="9fb36a9be1fea939f88617a85ea1a76432d1f6fdcc733ba133ec195729fafc24" exitCode=0
Jul 02 14:28:56.123715 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:56.123543    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj"
Jul 02 14:28:56.123715 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:56.123582    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj" event={"ID":"5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52","Type":"ContainerDied","Data":"9fb36a9be1fea939f88617a85ea1a76432d1f6fdcc733ba133ec195729fafc24"}
Jul 02 14:28:56.123715 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:56.123631    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj" event={"ID":"5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52","Type":"ContainerDied","Data":"fe143ec7f938e6f2853ba6fc59d80f6f897011f3a95f4531828d4662252d54e2"}
Jul 02 14:28:56.123715 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:56.123654    2583 scope.go:117] "RemoveContainer" containerID="9fb36a9be1fea939f88617a85ea1a76432d1f6fdcc733ba133ec195729fafc24"
Jul 02 14:28:56.132787 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:56.132767    2583 scope.go:117] "RemoveContainer" containerID="9fb36a9be1fea939f88617a85ea1a76432d1f6fdcc733ba133ec195729fafc24"
Jul 02 14:28:56.133101 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:28:56.133077    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fb36a9be1fea939f88617a85ea1a76432d1f6fdcc733ba133ec195729fafc24\": container with ID starting with 9fb36a9be1fea939f88617a85ea1a76432d1f6fdcc733ba133ec195729fafc24 not found: ID does not exist" containerID="9fb36a9be1fea939f88617a85ea1a76432d1f6fdcc733ba133ec195729fafc24"
Jul 02 14:28:56.133148 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:56.133110    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fb36a9be1fea939f88617a85ea1a76432d1f6fdcc733ba133ec195729fafc24"} err="failed to get container status \"9fb36a9be1fea939f88617a85ea1a76432d1f6fdcc733ba133ec195729fafc24\": rpc error: code = NotFound desc = could not find container \"9fb36a9be1fea939f88617a85ea1a76432d1f6fdcc733ba133ec195729fafc24\": container with ID starting with 9fb36a9be1fea939f88617a85ea1a76432d1f6fdcc733ba133ec195729fafc24 not found: ID does not exist"
Jul 02 14:28:56.146988 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:56.146939    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj"]
Jul 02 14:28:56.149017 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:56.148993    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-59r5rrj"]
Jul 02 14:28:57.233748 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:28:57.233648    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52" path="/var/lib/kubelet/pods/5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52/volumes"
Jul 02 14:29:02.205207 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.205170    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"]
Jul 02 14:29:02.205828 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.205805    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="131a76e2-7e0d-4225-84f2-6f7e2e445407" containerName="storage-initializer"
Jul 02 14:29:02.205893 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.205833    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="131a76e2-7e0d-4225-84f2-6f7e2e445407" containerName="storage-initializer"
Jul 02 14:29:02.205893 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.205848    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52" containerName="main"
Jul 02 14:29:02.205893 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.205856    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52" containerName="main"
Jul 02 14:29:02.205988 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.205899    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="131a76e2-7e0d-4225-84f2-6f7e2e445407" containerName="main"
Jul 02 14:29:02.205988 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.205907    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="131a76e2-7e0d-4225-84f2-6f7e2e445407" containerName="main"
Jul 02 14:29:02.206054 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.206002    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c8cd6b4-0901-49c6-9eb7-b2c9e6439c52" containerName="main"
Jul 02 14:29:02.206054 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.206016    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="131a76e2-7e0d-4225-84f2-6f7e2e445407" containerName="main"
Jul 02 14:29:02.210814 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.210793    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"
Jul 02 14:29:02.213615 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.213591    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 14:29:02.213736 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.213591    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 02 14:29:02.219612 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.219583    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"]
Jul 02 14:29:02.241409 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.241334    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:29:02.352339 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.352292    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54604b57-049c-4ed0-bdd3-cb8a1499218a-tls-certs\") pod \"tls-verification-test-kserve-cbc87bf9d-g9hvx\" (UID: \"54604b57-049c-4ed0-bdd3-cb8a1499218a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"
Jul 02 14:29:02.352557 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.352364    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54604b57-049c-4ed0-bdd3-cb8a1499218a-dshm\") pod \"tls-verification-test-kserve-cbc87bf9d-g9hvx\" (UID: \"54604b57-049c-4ed0-bdd3-cb8a1499218a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"
Jul 02 14:29:02.352557 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.352465    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54604b57-049c-4ed0-bdd3-cb8a1499218a-tmp-dir\") pod \"tls-verification-test-kserve-cbc87bf9d-g9hvx\" (UID: \"54604b57-049c-4ed0-bdd3-cb8a1499218a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"
Jul 02 14:29:02.352557 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.352496    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54604b57-049c-4ed0-bdd3-cb8a1499218a-kserve-provision-location\") pod \"tls-verification-test-kserve-cbc87bf9d-g9hvx\" (UID: \"54604b57-049c-4ed0-bdd3-cb8a1499218a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"
Jul 02 14:29:02.352557 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.352541    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54604b57-049c-4ed0-bdd3-cb8a1499218a-home\") pod \"tls-verification-test-kserve-cbc87bf9d-g9hvx\" (UID: \"54604b57-049c-4ed0-bdd3-cb8a1499218a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"
Jul 02 14:29:02.352775 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.352608    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzlkw\" (UniqueName: \"kubernetes.io/projected/54604b57-049c-4ed0-bdd3-cb8a1499218a-kube-api-access-bzlkw\") pod \"tls-verification-test-kserve-cbc87bf9d-g9hvx\" (UID: \"54604b57-049c-4ed0-bdd3-cb8a1499218a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"
Jul 02 14:29:02.352775 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.352695    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54604b57-049c-4ed0-bdd3-cb8a1499218a-model-cache\") pod \"tls-verification-test-kserve-cbc87bf9d-g9hvx\" (UID: \"54604b57-049c-4ed0-bdd3-cb8a1499218a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"
Jul 02 14:29:02.454050 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.454009    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54604b57-049c-4ed0-bdd3-cb8a1499218a-home\") pod \"tls-verification-test-kserve-cbc87bf9d-g9hvx\" (UID: \"54604b57-049c-4ed0-bdd3-cb8a1499218a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"
Jul 02 14:29:02.454254 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.454072    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzlkw\" (UniqueName: \"kubernetes.io/projected/54604b57-049c-4ed0-bdd3-cb8a1499218a-kube-api-access-bzlkw\") pod \"tls-verification-test-kserve-cbc87bf9d-g9hvx\" (UID: \"54604b57-049c-4ed0-bdd3-cb8a1499218a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"
Jul 02 14:29:02.454254 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.454099    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54604b57-049c-4ed0-bdd3-cb8a1499218a-model-cache\") pod \"tls-verification-test-kserve-cbc87bf9d-g9hvx\" (UID: \"54604b57-049c-4ed0-bdd3-cb8a1499218a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"
Jul 02 14:29:02.454254 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.454167    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54604b57-049c-4ed0-bdd3-cb8a1499218a-tls-certs\") pod \"tls-verification-test-kserve-cbc87bf9d-g9hvx\" (UID: \"54604b57-049c-4ed0-bdd3-cb8a1499218a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"
Jul 02 14:29:02.454254 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.454210    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54604b57-049c-4ed0-bdd3-cb8a1499218a-dshm\") pod \"tls-verification-test-kserve-cbc87bf9d-g9hvx\" (UID: \"54604b57-049c-4ed0-bdd3-cb8a1499218a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"
Jul 02 14:29:02.454254 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.454249    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54604b57-049c-4ed0-bdd3-cb8a1499218a-tmp-dir\") pod \"tls-verification-test-kserve-cbc87bf9d-g9hvx\" (UID: \"54604b57-049c-4ed0-bdd3-cb8a1499218a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"
Jul 02 14:29:02.454581 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.454379    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54604b57-049c-4ed0-bdd3-cb8a1499218a-kserve-provision-location\") pod \"tls-verification-test-kserve-cbc87bf9d-g9hvx\" (UID: \"54604b57-049c-4ed0-bdd3-cb8a1499218a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"
Jul 02 14:29:02.454581 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.454449    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54604b57-049c-4ed0-bdd3-cb8a1499218a-home\") pod \"tls-verification-test-kserve-cbc87bf9d-g9hvx\" (UID: \"54604b57-049c-4ed0-bdd3-cb8a1499218a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"
Jul 02 14:29:02.454581 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.454510    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54604b57-049c-4ed0-bdd3-cb8a1499218a-model-cache\") pod \"tls-verification-test-kserve-cbc87bf9d-g9hvx\" (UID: \"54604b57-049c-4ed0-bdd3-cb8a1499218a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"
Jul 02 14:29:02.454699 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.454579    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54604b57-049c-4ed0-bdd3-cb8a1499218a-tmp-dir\") pod \"tls-verification-test-kserve-cbc87bf9d-g9hvx\" (UID: \"54604b57-049c-4ed0-bdd3-cb8a1499218a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"
Jul 02 14:29:02.454737 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.454710    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54604b57-049c-4ed0-bdd3-cb8a1499218a-kserve-provision-location\") pod \"tls-verification-test-kserve-cbc87bf9d-g9hvx\" (UID: \"54604b57-049c-4ed0-bdd3-cb8a1499218a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"
Jul 02 14:29:02.456510 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.456461    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54604b57-049c-4ed0-bdd3-cb8a1499218a-dshm\") pod \"tls-verification-test-kserve-cbc87bf9d-g9hvx\" (UID: \"54604b57-049c-4ed0-bdd3-cb8a1499218a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"
Jul 02 14:29:02.456723 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.456706    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54604b57-049c-4ed0-bdd3-cb8a1499218a-tls-certs\") pod \"tls-verification-test-kserve-cbc87bf9d-g9hvx\" (UID: \"54604b57-049c-4ed0-bdd3-cb8a1499218a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"
Jul 02 14:29:02.462321 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.462301    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzlkw\" (UniqueName: \"kubernetes.io/projected/54604b57-049c-4ed0-bdd3-cb8a1499218a-kube-api-access-bzlkw\") pod \"tls-verification-test-kserve-cbc87bf9d-g9hvx\" (UID: \"54604b57-049c-4ed0-bdd3-cb8a1499218a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"
Jul 02 14:29:02.515630 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.515593    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz"]
Jul 02 14:29:02.519374 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.519355    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz"
Jul 02 14:29:02.521943 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.521889    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"
Jul 02 14:29:02.522109 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.522088    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"tls-verification-test-epp-sa-dockercfg-bscvx\""
Jul 02 14:29:02.529806 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.529770    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz"]
Jul 02 14:29:02.541735 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.541700    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:29:02.656195 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.656157    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx9vt\" (UniqueName: \"kubernetes.io/projected/bce44fe0-3470-4f9d-bf94-487cfb93477d-kube-api-access-kx9vt\") pod \"tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz\" (UID: \"bce44fe0-3470-4f9d-bf94-487cfb93477d\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz"
Jul 02 14:29:02.656383 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.656205    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bce44fe0-3470-4f9d-bf94-487cfb93477d-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz\" (UID: \"bce44fe0-3470-4f9d-bf94-487cfb93477d\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz"
Jul 02 14:29:02.656383 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.656236    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bce44fe0-3470-4f9d-bf94-487cfb93477d-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz\" (UID: \"bce44fe0-3470-4f9d-bf94-487cfb93477d\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz"
Jul 02 14:29:02.656383 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.656345    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bce44fe0-3470-4f9d-bf94-487cfb93477d-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz\" (UID: \"bce44fe0-3470-4f9d-bf94-487cfb93477d\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz"
Jul 02 14:29:02.666695 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.666669    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"]
Jul 02 14:29:02.669093 ip-10-0-135-143 kubenswrapper[2583]: W0702 14:29:02.669062    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54604b57_049c_4ed0_bdd3_cb8a1499218a.slice/crio-40765db642f8c7886f43e6307cb976d285474903316426bd4dcaf784e3caa9cc WatchSource:0}: Error finding container 40765db642f8c7886f43e6307cb976d285474903316426bd4dcaf784e3caa9cc: Status 404 returned error can't find the container with id 40765db642f8c7886f43e6307cb976d285474903316426bd4dcaf784e3caa9cc
Jul 02 14:29:02.757933 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.757891    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bce44fe0-3470-4f9d-bf94-487cfb93477d-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz\" (UID: \"bce44fe0-3470-4f9d-bf94-487cfb93477d\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz"
Jul 02 14:29:02.758142 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.757998    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bce44fe0-3470-4f9d-bf94-487cfb93477d-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz\" (UID: \"bce44fe0-3470-4f9d-bf94-487cfb93477d\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz"
Jul 02 14:29:02.758142 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.758063    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx9vt\" (UniqueName: \"kubernetes.io/projected/bce44fe0-3470-4f9d-bf94-487cfb93477d-kube-api-access-kx9vt\") pod \"tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz\" (UID: \"bce44fe0-3470-4f9d-bf94-487cfb93477d\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz"
Jul 02 14:29:02.758142 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.758107    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bce44fe0-3470-4f9d-bf94-487cfb93477d-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz\" (UID: \"bce44fe0-3470-4f9d-bf94-487cfb93477d\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz"
Jul 02 14:29:02.758361 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.758336    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bce44fe0-3470-4f9d-bf94-487cfb93477d-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz\" (UID: \"bce44fe0-3470-4f9d-bf94-487cfb93477d\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz"
Jul 02 14:29:02.758465 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.758361    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bce44fe0-3470-4f9d-bf94-487cfb93477d-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz\" (UID: \"bce44fe0-3470-4f9d-bf94-487cfb93477d\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz"
Jul 02 14:29:02.760654 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.760629    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bce44fe0-3470-4f9d-bf94-487cfb93477d-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz\" (UID: \"bce44fe0-3470-4f9d-bf94-487cfb93477d\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz"
Jul 02 14:29:02.766442 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.766376    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx9vt\" (UniqueName: \"kubernetes.io/projected/bce44fe0-3470-4f9d-bf94-487cfb93477d-kube-api-access-kx9vt\") pod \"tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz\" (UID: \"bce44fe0-3470-4f9d-bf94-487cfb93477d\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz"
Jul 02 14:29:02.830504 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.830460    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz"
Jul 02 14:29:02.981034 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:02.981009    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz"]
Jul 02 14:29:02.984074 ip-10-0-135-143 kubenswrapper[2583]: W0702 14:29:02.984039    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbce44fe0_3470_4f9d_bf94_487cfb93477d.slice/crio-c564f4185f838c989a677510da514eecea0f0513504edd45da11da9ae5886fbf WatchSource:0}: Error finding container c564f4185f838c989a677510da514eecea0f0513504edd45da11da9ae5886fbf: Status 404 returned error can't find the container with id c564f4185f838c989a677510da514eecea0f0513504edd45da11da9ae5886fbf
Jul 02 14:29:03.161020 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:03.160978    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz" event={"ID":"bce44fe0-3470-4f9d-bf94-487cfb93477d","Type":"ContainerStarted","Data":"345c604bdb58cf2eeee152c70c8e0eee3ebc51636fe60fb05da82f7f6b88259c"}
Jul 02 14:29:03.161224 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:03.161057    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz" event={"ID":"bce44fe0-3470-4f9d-bf94-487cfb93477d","Type":"ContainerStarted","Data":"c564f4185f838c989a677510da514eecea0f0513504edd45da11da9ae5886fbf"}
Jul 02 14:29:03.162739 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:03.162706    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx" event={"ID":"54604b57-049c-4ed0-bdd3-cb8a1499218a","Type":"ContainerStarted","Data":"86ef3ae669e7d6af274813ef1a84480944519423f8dba3fc49cdf69ce57eef97"}
Jul 02 14:29:03.162872 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:03.162754    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx" event={"ID":"54604b57-049c-4ed0-bdd3-cb8a1499218a","Type":"ContainerStarted","Data":"40765db642f8c7886f43e6307cb976d285474903316426bd4dcaf784e3caa9cc"}
Jul 02 14:29:04.169184 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:04.169145    2583 generic.go:356] "Generic (PLEG): container finished" podID="bce44fe0-3470-4f9d-bf94-487cfb93477d" containerID="345c604bdb58cf2eeee152c70c8e0eee3ebc51636fe60fb05da82f7f6b88259c" exitCode=0
Jul 02 14:29:04.169638 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:04.169235    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz" event={"ID":"bce44fe0-3470-4f9d-bf94-487cfb93477d","Type":"ContainerDied","Data":"345c604bdb58cf2eeee152c70c8e0eee3ebc51636fe60fb05da82f7f6b88259c"}
Jul 02 14:29:05.176781 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:05.176746    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz" event={"ID":"bce44fe0-3470-4f9d-bf94-487cfb93477d","Type":"ContainerStarted","Data":"9dacfb2946dbf34448474803b344ced5f891da56ff7dbf4000b8be6d035c95fc"}
Jul 02 14:29:05.177169 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:05.176849    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz"
Jul 02 14:29:05.205956 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:05.205884    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz" podStartSLOduration=2.450376241 podStartE2EDuration="3.205860594s" totalImagesPullingTime="0s" totalInitContainerRuntime="755.484353ms" isStatefulPod=false podCreationTimestamp="2026-07-02 14:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:29:05.202746809 +0000 UTC m=+4456.430657986" watchObservedRunningTime="2026-07-02 14:29:05.205860594 +0000 UTC m=+4456.433771769"
Jul 02 14:29:07.190542 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:07.190502    2583 generic.go:356] "Generic (PLEG): container finished" podID="54604b57-049c-4ed0-bdd3-cb8a1499218a" containerID="86ef3ae669e7d6af274813ef1a84480944519423f8dba3fc49cdf69ce57eef97" exitCode=0
Jul 02 14:29:07.191055 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:07.190566    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx" event={"ID":"54604b57-049c-4ed0-bdd3-cb8a1499218a","Type":"ContainerDied","Data":"86ef3ae669e7d6af274813ef1a84480944519423f8dba3fc49cdf69ce57eef97"}
Jul 02 14:29:08.198572 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:08.198533    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx" event={"ID":"54604b57-049c-4ed0-bdd3-cb8a1499218a","Type":"ContainerStarted","Data":"5b52970088ff888f0837fceefcaaa1b39c255b028ff2b2b2222850e8dc1ce324"}
Jul 02 14:29:08.223612 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:08.223556    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx" podStartSLOduration=1.951747894 podStartE2EDuration="6.223536825s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.271788931s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:29:08.221381778 +0000 UTC m=+4459.449292945" watchObservedRunningTime="2026-07-02 14:29:08.223536825 +0000 UTC m=+4459.451447998"
Jul 02 14:29:08.522380 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:08.522296    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"
Jul 02 14:29:10.428625 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:10.428586    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl" podUID="89360742-358c-4308-b211-e677a5fa700f" containerName="main" containerID="cri-o://cc7cb59c8f681f6305c26f684fe51ec5d15f0a236a418f1e42fa6b4fcf35c3ee" gracePeriod=45
Jul 02 14:29:10.547413 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:10.547330    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl" podUID="89360742-358c-4308-b211-e677a5fa700f" containerName="llm-d-routing-sidecar" containerID="cri-o://1f792493590971ed5a224762b3f1e9be85d3c67b98b7f1a7573c9d060a1d0d61" gracePeriod=45
Jul 02 14:29:10.816142 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:10.816114    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"
Jul 02 14:29:10.940822 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:10.940778    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89360742-358c-4308-b211-e677a5fa700f-home\") pod \"89360742-358c-4308-b211-e677a5fa700f\" (UID: \"89360742-358c-4308-b211-e677a5fa700f\") "
Jul 02 14:29:10.941018 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:10.940833    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89360742-358c-4308-b211-e677a5fa700f-tls-certs\") pod \"89360742-358c-4308-b211-e677a5fa700f\" (UID: \"89360742-358c-4308-b211-e677a5fa700f\") "
Jul 02 14:29:10.941018 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:10.940905    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89360742-358c-4308-b211-e677a5fa700f-model-cache\") pod \"89360742-358c-4308-b211-e677a5fa700f\" (UID: \"89360742-358c-4308-b211-e677a5fa700f\") "
Jul 02 14:29:10.941018 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:10.940955    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89360742-358c-4308-b211-e677a5fa700f-dshm\") pod \"89360742-358c-4308-b211-e677a5fa700f\" (UID: \"89360742-358c-4308-b211-e677a5fa700f\") "
Jul 02 14:29:10.941018 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:10.940984    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89360742-358c-4308-b211-e677a5fa700f-tmp-dir\") pod \"89360742-358c-4308-b211-e677a5fa700f\" (UID: \"89360742-358c-4308-b211-e677a5fa700f\") "
Jul 02 14:29:10.941018 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:10.941018    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vc7t\" (UniqueName: \"kubernetes.io/projected/89360742-358c-4308-b211-e677a5fa700f-kube-api-access-9vc7t\") pod \"89360742-358c-4308-b211-e677a5fa700f\" (UID: \"89360742-358c-4308-b211-e677a5fa700f\") "
Jul 02 14:29:10.941288 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:10.941114    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89360742-358c-4308-b211-e677a5fa700f-home" (OuterVolumeSpecName: "home") pod "89360742-358c-4308-b211-e677a5fa700f" (UID: "89360742-358c-4308-b211-e677a5fa700f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:29:10.941288 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:10.941214    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89360742-358c-4308-b211-e677a5fa700f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "89360742-358c-4308-b211-e677a5fa700f" (UID: "89360742-358c-4308-b211-e677a5fa700f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:29:10.941288 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:10.941224    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89360742-358c-4308-b211-e677a5fa700f-model-cache" (OuterVolumeSpecName: "model-cache") pod "89360742-358c-4308-b211-e677a5fa700f" (UID: "89360742-358c-4308-b211-e677a5fa700f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:29:10.941418 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:10.941378    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89360742-358c-4308-b211-e677a5fa700f-home\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:29:10.941418 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:10.941415    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89360742-358c-4308-b211-e677a5fa700f-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:29:10.941497 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:10.941430    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89360742-358c-4308-b211-e677a5fa700f-tmp-dir\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:29:10.943233 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:10.943206    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89360742-358c-4308-b211-e677a5fa700f-kube-api-access-9vc7t" (OuterVolumeSpecName: "kube-api-access-9vc7t") pod "89360742-358c-4308-b211-e677a5fa700f" (UID: "89360742-358c-4308-b211-e677a5fa700f"). InnerVolumeSpecName "kube-api-access-9vc7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:29:10.943568 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:10.943546    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89360742-358c-4308-b211-e677a5fa700f-dshm" (OuterVolumeSpecName: "dshm") pod "89360742-358c-4308-b211-e677a5fa700f" (UID: "89360742-358c-4308-b211-e677a5fa700f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:29:10.943568 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:10.943558    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89360742-358c-4308-b211-e677a5fa700f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "89360742-358c-4308-b211-e677a5fa700f" (UID: "89360742-358c-4308-b211-e677a5fa700f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:29:11.042679 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:11.042590    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vc7t\" (UniqueName: \"kubernetes.io/projected/89360742-358c-4308-b211-e677a5fa700f-kube-api-access-9vc7t\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:29:11.042679 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:11.042622    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89360742-358c-4308-b211-e677a5fa700f-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:29:11.042679 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:11.042631    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89360742-358c-4308-b211-e677a5fa700f-dshm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:29:11.219319 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:11.219277    2583 generic.go:356] "Generic (PLEG): container finished" podID="89360742-358c-4308-b211-e677a5fa700f" containerID="cc7cb59c8f681f6305c26f684fe51ec5d15f0a236a418f1e42fa6b4fcf35c3ee" exitCode=0
Jul 02 14:29:11.219319 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:11.219313    2583 generic.go:356] "Generic (PLEG): container finished" podID="89360742-358c-4308-b211-e677a5fa700f" containerID="1f792493590971ed5a224762b3f1e9be85d3c67b98b7f1a7573c9d060a1d0d61" exitCode=0
Jul 02 14:29:11.219542 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:11.219339    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"
Jul 02 14:29:11.219542 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:11.219364    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl" event={"ID":"89360742-358c-4308-b211-e677a5fa700f","Type":"ContainerDied","Data":"cc7cb59c8f681f6305c26f684fe51ec5d15f0a236a418f1e42fa6b4fcf35c3ee"}
Jul 02 14:29:11.219542 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:11.219468    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl" event={"ID":"89360742-358c-4308-b211-e677a5fa700f","Type":"ContainerDied","Data":"1f792493590971ed5a224762b3f1e9be85d3c67b98b7f1a7573c9d060a1d0d61"}
Jul 02 14:29:11.219542 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:11.219496    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl" event={"ID":"89360742-358c-4308-b211-e677a5fa700f","Type":"ContainerDied","Data":"af11b37bfd08d9927af8cf6881e57184abfab61ba815cb8f67227934babd7948"}
Jul 02 14:29:11.219542 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:11.219526    2583 scope.go:117] "RemoveContainer" containerID="cc7cb59c8f681f6305c26f684fe51ec5d15f0a236a418f1e42fa6b4fcf35c3ee"
Jul 02 14:29:11.228883 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:11.228851    2583 scope.go:117] "RemoveContainer" containerID="1f792493590971ed5a224762b3f1e9be85d3c67b98b7f1a7573c9d060a1d0d61"
Jul 02 14:29:11.239431 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:11.239381    2583 scope.go:117] "RemoveContainer" containerID="cc7cb59c8f681f6305c26f684fe51ec5d15f0a236a418f1e42fa6b4fcf35c3ee"
Jul 02 14:29:11.239751 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:29:11.239729    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc7cb59c8f681f6305c26f684fe51ec5d15f0a236a418f1e42fa6b4fcf35c3ee\": container with ID starting with cc7cb59c8f681f6305c26f684fe51ec5d15f0a236a418f1e42fa6b4fcf35c3ee not found: ID does not exist" containerID="cc7cb59c8f681f6305c26f684fe51ec5d15f0a236a418f1e42fa6b4fcf35c3ee"
Jul 02 14:29:11.239823 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:11.239759    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc7cb59c8f681f6305c26f684fe51ec5d15f0a236a418f1e42fa6b4fcf35c3ee"} err="failed to get container status \"cc7cb59c8f681f6305c26f684fe51ec5d15f0a236a418f1e42fa6b4fcf35c3ee\": rpc error: code = NotFound desc = could not find container \"cc7cb59c8f681f6305c26f684fe51ec5d15f0a236a418f1e42fa6b4fcf35c3ee\": container with ID starting with cc7cb59c8f681f6305c26f684fe51ec5d15f0a236a418f1e42fa6b4fcf35c3ee not found: ID does not exist"
Jul 02 14:29:11.239823 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:11.239779    2583 scope.go:117] "RemoveContainer" containerID="1f792493590971ed5a224762b3f1e9be85d3c67b98b7f1a7573c9d060a1d0d61"
Jul 02 14:29:11.240108 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:29:11.240077    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f792493590971ed5a224762b3f1e9be85d3c67b98b7f1a7573c9d060a1d0d61\": container with ID starting with 1f792493590971ed5a224762b3f1e9be85d3c67b98b7f1a7573c9d060a1d0d61 not found: ID does not exist" containerID="1f792493590971ed5a224762b3f1e9be85d3c67b98b7f1a7573c9d060a1d0d61"
Jul 02 14:29:11.240108 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:11.240131    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f792493590971ed5a224762b3f1e9be85d3c67b98b7f1a7573c9d060a1d0d61"} err="failed to get container status \"1f792493590971ed5a224762b3f1e9be85d3c67b98b7f1a7573c9d060a1d0d61\": rpc error: code = NotFound desc = could not find container \"1f792493590971ed5a224762b3f1e9be85d3c67b98b7f1a7573c9d060a1d0d61\": container with ID starting with 1f792493590971ed5a224762b3f1e9be85d3c67b98b7f1a7573c9d060a1d0d61 not found: ID does not exist"
Jul 02 14:29:11.240108 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:11.240164    2583 scope.go:117] "RemoveContainer" containerID="cc7cb59c8f681f6305c26f684fe51ec5d15f0a236a418f1e42fa6b4fcf35c3ee"
Jul 02 14:29:11.240680 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:11.240597    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc7cb59c8f681f6305c26f684fe51ec5d15f0a236a418f1e42fa6b4fcf35c3ee"} err="failed to get container status \"cc7cb59c8f681f6305c26f684fe51ec5d15f0a236a418f1e42fa6b4fcf35c3ee\": rpc error: code = NotFound desc = could not find container \"cc7cb59c8f681f6305c26f684fe51ec5d15f0a236a418f1e42fa6b4fcf35c3ee\": container with ID starting with cc7cb59c8f681f6305c26f684fe51ec5d15f0a236a418f1e42fa6b4fcf35c3ee not found: ID does not exist"
Jul 02 14:29:11.240680 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:11.240623    2583 scope.go:117] "RemoveContainer" containerID="1f792493590971ed5a224762b3f1e9be85d3c67b98b7f1a7573c9d060a1d0d61"
Jul 02 14:29:11.240980 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:11.240947    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f792493590971ed5a224762b3f1e9be85d3c67b98b7f1a7573c9d060a1d0d61"} err="failed to get container status \"1f792493590971ed5a224762b3f1e9be85d3c67b98b7f1a7573c9d060a1d0d61\": rpc error: code = NotFound desc = could not find container \"1f792493590971ed5a224762b3f1e9be85d3c67b98b7f1a7573c9d060a1d0d61\": container with ID starting with 1f792493590971ed5a224762b3f1e9be85d3c67b98b7f1a7573c9d060a1d0d61 not found: ID does not exist"
Jul 02 14:29:11.243007 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:11.242986    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"]
Jul 02 14:29:11.249101 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:11.249072    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-7qwnl"]
Jul 02 14:29:12.522099 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:12.522061    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"
Jul 02 14:29:12.523811 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:12.523782    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx" podUID="54604b57-049c-4ed0-bdd3-cb8a1499218a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Jul 02 14:29:13.234518 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:13.234483    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89360742-358c-4308-b211-e677a5fa700f" path="/var/lib/kubelet/pods/89360742-358c-4308-b211-e677a5fa700f/volumes"
Jul 02 14:29:22.522549 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:22.522506    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx" podUID="54604b57-049c-4ed0-bdd3-cb8a1499218a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Jul 02 14:29:25.693917 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:25.693881    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:29:32.522312 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:32.522261    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx" podUID="54604b57-049c-4ed0-bdd3-cb8a1499218a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Jul 02 14:29:33.416038 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:33.416003    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:29:36.185315 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:36.185284    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz"
Jul 02 14:29:42.522775 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:42.522661    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx" podUID="54604b57-049c-4ed0-bdd3-cb8a1499218a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Jul 02 14:29:43.329715 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.329680    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb"]
Jul 02 14:29:43.330136 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.330120    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89360742-358c-4308-b211-e677a5fa700f" containerName="main"
Jul 02 14:29:43.330136 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.330136    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="89360742-358c-4308-b211-e677a5fa700f" containerName="main"
Jul 02 14:29:43.330308 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.330157    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89360742-358c-4308-b211-e677a5fa700f" containerName="llm-d-routing-sidecar"
Jul 02 14:29:43.330308 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.330163    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="89360742-358c-4308-b211-e677a5fa700f" containerName="llm-d-routing-sidecar"
Jul 02 14:29:43.330308 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.330232    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="89360742-358c-4308-b211-e677a5fa700f" containerName="main"
Jul 02 14:29:43.330308 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.330246    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="89360742-358c-4308-b211-e677a5fa700f" containerName="llm-d-routing-sidecar"
Jul 02 14:29:43.332846 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.332820    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb"
Jul 02 14:29:43.336510 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.336484    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 02 14:29:43.346214 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.346182    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb"]
Jul 02 14:29:43.373279 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.373245    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:29:43.471627 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.471587    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhxjh\" (UniqueName: \"kubernetes.io/projected/24b229e5-d266-43b2-aaa2-4918404b5063-kube-api-access-bhxjh\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb\" (UID: \"24b229e5-d266-43b2-aaa2-4918404b5063\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb"
Jul 02 14:29:43.471804 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.471641    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24b229e5-d266-43b2-aaa2-4918404b5063-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb\" (UID: \"24b229e5-d266-43b2-aaa2-4918404b5063\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb"
Jul 02 14:29:43.471804 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.471669    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24b229e5-d266-43b2-aaa2-4918404b5063-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb\" (UID: \"24b229e5-d266-43b2-aaa2-4918404b5063\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb"
Jul 02 14:29:43.471804 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.471716    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24b229e5-d266-43b2-aaa2-4918404b5063-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb\" (UID: \"24b229e5-d266-43b2-aaa2-4918404b5063\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb"
Jul 02 14:29:43.471919 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.471835    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24b229e5-d266-43b2-aaa2-4918404b5063-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb\" (UID: \"24b229e5-d266-43b2-aaa2-4918404b5063\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb"
Jul 02 14:29:43.471919 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.471865    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24b229e5-d266-43b2-aaa2-4918404b5063-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb\" (UID: \"24b229e5-d266-43b2-aaa2-4918404b5063\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb"
Jul 02 14:29:43.558337 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.558293    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j"]
Jul 02 14:29:43.561498 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.561473    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j"
Jul 02 14:29:43.564118 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.564089    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-pgpnc\""
Jul 02 14:29:43.573172 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.573141    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhxjh\" (UniqueName: \"kubernetes.io/projected/24b229e5-d266-43b2-aaa2-4918404b5063-kube-api-access-bhxjh\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb\" (UID: \"24b229e5-d266-43b2-aaa2-4918404b5063\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb"
Jul 02 14:29:43.573338 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.573203    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24b229e5-d266-43b2-aaa2-4918404b5063-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb\" (UID: \"24b229e5-d266-43b2-aaa2-4918404b5063\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb"
Jul 02 14:29:43.573338 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.573229    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24b229e5-d266-43b2-aaa2-4918404b5063-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb\" (UID: \"24b229e5-d266-43b2-aaa2-4918404b5063\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb"
Jul 02 14:29:43.573338 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.573287    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24b229e5-d266-43b2-aaa2-4918404b5063-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb\" (UID: \"24b229e5-d266-43b2-aaa2-4918404b5063\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb"
Jul 02 14:29:43.573552 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.573369    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24b229e5-d266-43b2-aaa2-4918404b5063-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb\" (UID: \"24b229e5-d266-43b2-aaa2-4918404b5063\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb"
Jul 02 14:29:43.573552 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.573427    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24b229e5-d266-43b2-aaa2-4918404b5063-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb\" (UID: \"24b229e5-d266-43b2-aaa2-4918404b5063\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb"
Jul 02 14:29:43.573957 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.573846    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24b229e5-d266-43b2-aaa2-4918404b5063-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb\" (UID: \"24b229e5-d266-43b2-aaa2-4918404b5063\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb"
Jul 02 14:29:43.574071 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.574053    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24b229e5-d266-43b2-aaa2-4918404b5063-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb\" (UID: \"24b229e5-d266-43b2-aaa2-4918404b5063\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb"
Jul 02 14:29:43.574347 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.574324    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24b229e5-d266-43b2-aaa2-4918404b5063-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb\" (UID: \"24b229e5-d266-43b2-aaa2-4918404b5063\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb"
Jul 02 14:29:43.574495 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.574468    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j"]
Jul 02 14:29:43.576459 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.576379    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24b229e5-d266-43b2-aaa2-4918404b5063-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb\" (UID: \"24b229e5-d266-43b2-aaa2-4918404b5063\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb"
Jul 02 14:29:43.576825 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.576800    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24b229e5-d266-43b2-aaa2-4918404b5063-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb\" (UID: \"24b229e5-d266-43b2-aaa2-4918404b5063\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb"
Jul 02 14:29:43.584955 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.584885    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhxjh\" (UniqueName: \"kubernetes.io/projected/24b229e5-d266-43b2-aaa2-4918404b5063-kube-api-access-bhxjh\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb\" (UID: \"24b229e5-d266-43b2-aaa2-4918404b5063\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb"
Jul 02 14:29:43.612689 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.612649    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:29:43.647507 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.647455    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb"
Jul 02 14:29:43.674637 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.674599    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a8d760a6-4b8d-462b-b797-75efb61912ad-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j\" (UID: \"a8d760a6-4b8d-462b-b797-75efb61912ad\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j"
Jul 02 14:29:43.674835 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.674656    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8d760a6-4b8d-462b-b797-75efb61912ad-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j\" (UID: \"a8d760a6-4b8d-462b-b797-75efb61912ad\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j"
Jul 02 14:29:43.674835 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.674785    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdktn\" (UniqueName: \"kubernetes.io/projected/a8d760a6-4b8d-462b-b797-75efb61912ad-kube-api-access-cdktn\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j\" (UID: \"a8d760a6-4b8d-462b-b797-75efb61912ad\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j"
Jul 02 14:29:43.776451 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.776412    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a8d760a6-4b8d-462b-b797-75efb61912ad-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j\" (UID: \"a8d760a6-4b8d-462b-b797-75efb61912ad\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j"
Jul 02 14:29:43.776637 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.776476    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8d760a6-4b8d-462b-b797-75efb61912ad-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j\" (UID: \"a8d760a6-4b8d-462b-b797-75efb61912ad\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j"
Jul 02 14:29:43.776637 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.776549    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdktn\" (UniqueName: \"kubernetes.io/projected/a8d760a6-4b8d-462b-b797-75efb61912ad-kube-api-access-cdktn\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j\" (UID: \"a8d760a6-4b8d-462b-b797-75efb61912ad\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j"
Jul 02 14:29:43.776802 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.776776    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a8d760a6-4b8d-462b-b797-75efb61912ad-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j\" (UID: \"a8d760a6-4b8d-462b-b797-75efb61912ad\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j"
Jul 02 14:29:43.779499 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.779471    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8d760a6-4b8d-462b-b797-75efb61912ad-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j\" (UID: \"a8d760a6-4b8d-462b-b797-75efb61912ad\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j"
Jul 02 14:29:43.784770 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.784738    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdktn\" (UniqueName: \"kubernetes.io/projected/a8d760a6-4b8d-462b-b797-75efb61912ad-kube-api-access-cdktn\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j\" (UID: \"a8d760a6-4b8d-462b-b797-75efb61912ad\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j"
Jul 02 14:29:43.793935 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.793899    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb"]
Jul 02 14:29:43.794953 ip-10-0-135-143 kubenswrapper[2583]: W0702 14:29:43.794929    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b229e5_d266_43b2_aaa2_4918404b5063.slice/crio-da058fa648c817c4408a60094f72cc0b87623ea2d2d519213a7ee9b86433b9f5 WatchSource:0}: Error finding container da058fa648c817c4408a60094f72cc0b87623ea2d2d519213a7ee9b86433b9f5: Status 404 returned error can't find the container with id da058fa648c817c4408a60094f72cc0b87623ea2d2d519213a7ee9b86433b9f5
Jul 02 14:29:43.872364 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:43.872332    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j"
Jul 02 14:29:44.018971 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:44.018923    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j"]
Jul 02 14:29:44.024437 ip-10-0-135-143 kubenswrapper[2583]: W0702 14:29:44.023774    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d760a6_4b8d_462b_b797_75efb61912ad.slice/crio-aeadf6332d7ec1f17629737e3306c8ed1307ca63d12af622a6d3de95e8780e24 WatchSource:0}: Error finding container aeadf6332d7ec1f17629737e3306c8ed1307ca63d12af622a6d3de95e8780e24: Status 404 returned error can't find the container with id aeadf6332d7ec1f17629737e3306c8ed1307ca63d12af622a6d3de95e8780e24
Jul 02 14:29:44.409444 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:44.409378    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j" event={"ID":"a8d760a6-4b8d-462b-b797-75efb61912ad","Type":"ContainerStarted","Data":"a7e9d36a6af790e46c609b97a49b967705edb86a2e0f7c972fea5308fbfa9498"}
Jul 02 14:29:44.409656 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:44.409464    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j" event={"ID":"a8d760a6-4b8d-462b-b797-75efb61912ad","Type":"ContainerStarted","Data":"aeadf6332d7ec1f17629737e3306c8ed1307ca63d12af622a6d3de95e8780e24"}
Jul 02 14:29:44.409656 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:44.409491    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j"
Jul 02 14:29:44.411110 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:44.411074    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb" event={"ID":"24b229e5-d266-43b2-aaa2-4918404b5063","Type":"ContainerStarted","Data":"4d3be208e4425d6be13b2ce1e3d2c522505afda4b35e7183e13bade8a3016a1d"}
Jul 02 14:29:44.411230 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:44.411120    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb" event={"ID":"24b229e5-d266-43b2-aaa2-4918404b5063","Type":"ContainerStarted","Data":"da058fa648c817c4408a60094f72cc0b87623ea2d2d519213a7ee9b86433b9f5"}
Jul 02 14:29:44.427251 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:44.427189    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j" podStartSLOduration=1.427168236 podStartE2EDuration="1.427168236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:29:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:29:44.426565894 +0000 UTC m=+4495.654477070" watchObservedRunningTime="2026-07-02 14:29:44.427168236 +0000 UTC m=+4495.655079412"
Jul 02 14:29:44.446433 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:44.446351    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb" podStartSLOduration=1.446331708 podStartE2EDuration="1.446331708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:29:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:29:44.44398311 +0000 UTC m=+4495.671894287" watchObservedRunningTime="2026-07-02 14:29:44.446331708 +0000 UTC m=+4495.674242882"
Jul 02 14:29:44.648342 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:44.648309    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb"
Jul 02 14:29:49.674463 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:49.674347    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 14:29:49.688073 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:49.688047    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 14:29:52.522748 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:52.522698    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx" podUID="54604b57-049c-4ed0-bdd3-cb8a1499218a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Jul 02 14:29:53.647948 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:53.647907    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb"
Jul 02 14:29:53.661076 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:53.661044    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb"
Jul 02 14:29:54.475425 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:29:54.475372    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb"
Jul 02 14:30:02.522753 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:30:02.522710    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx" podUID="54604b57-049c-4ed0-bdd3-cb8a1499218a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Jul 02 14:30:12.523109 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:30:12.523064    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx" podUID="54604b57-049c-4ed0-bdd3-cb8a1499218a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Jul 02 14:30:15.418676 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:30:15.418647    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j"
Jul 02 14:30:22.523150 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:30:22.523093    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx" podUID="54604b57-049c-4ed0-bdd3-cb8a1499218a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Jul 02 14:30:26.994179 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:30:26.994137    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:30:32.522788 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:30:32.522736    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx" podUID="54604b57-049c-4ed0-bdd3-cb8a1499218a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Jul 02 14:30:42.523320 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:30:42.523270    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx" podUID="54604b57-049c-4ed0-bdd3-cb8a1499218a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Jul 02 14:30:52.522893 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:30:52.522846    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx" podUID="54604b57-049c-4ed0-bdd3-cb8a1499218a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Jul 02 14:31:02.523024 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:02.522971    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx" podUID="54604b57-049c-4ed0-bdd3-cb8a1499218a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Jul 02 14:31:04.827584 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:04.827547    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb"]
Jul 02 14:31:04.831458 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:04.831431    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j"]
Jul 02 14:31:04.833216 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:04.833189    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:31:04.842607 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:31:04.842562    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 02 14:31:04.842607 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 14:31:04.842607 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 14:31:04.842607 ip-10-0-135-143 kubenswrapper[2583]:  > containerID="a7e9d36a6af790e46c609b97a49b967705edb86a2e0f7c972fea5308fbfa9498" cmd=["/bin/sleep","15"]
Jul 02 14:31:04.842973 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:04.842639    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j" message="" err=<
Jul 02 14:31:04.842973 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 14:31:04.842973 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 14:31:04.842973 ip-10-0-135-143 kubenswrapper[2583]:  >
Jul 02 14:31:04.842973 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:31:04.842658    2583 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 02 14:31:04.842973 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 14:31:04.842973 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 14:31:04.842973 ip-10-0-135-143 kubenswrapper[2583]:  > pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j" podUID="a8d760a6-4b8d-462b-b797-75efb61912ad" containerName="main" containerID="cri-o://a7e9d36a6af790e46c609b97a49b967705edb86a2e0f7c972fea5308fbfa9498"
Jul 02 14:31:04.842973 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:04.842703    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j" podUID="a8d760a6-4b8d-462b-b797-75efb61912ad" containerName="main" containerID="cri-o://a7e9d36a6af790e46c609b97a49b967705edb86a2e0f7c972fea5308fbfa9498" gracePeriod=60
Jul 02 14:31:05.096139 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:05.096098    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j"
Jul 02 14:31:05.197094 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:05.197058    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdktn\" (UniqueName: \"kubernetes.io/projected/a8d760a6-4b8d-462b-b797-75efb61912ad-kube-api-access-cdktn\") pod \"a8d760a6-4b8d-462b-b797-75efb61912ad\" (UID: \"a8d760a6-4b8d-462b-b797-75efb61912ad\") "
Jul 02 14:31:05.197310 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:05.197114    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a8d760a6-4b8d-462b-b797-75efb61912ad-tokenizer-uds\") pod \"a8d760a6-4b8d-462b-b797-75efb61912ad\" (UID: \"a8d760a6-4b8d-462b-b797-75efb61912ad\") "
Jul 02 14:31:05.197310 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:05.197231    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8d760a6-4b8d-462b-b797-75efb61912ad-tls-certs\") pod \"a8d760a6-4b8d-462b-b797-75efb61912ad\" (UID: \"a8d760a6-4b8d-462b-b797-75efb61912ad\") "
Jul 02 14:31:05.197507 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:05.197472    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d760a6-4b8d-462b-b797-75efb61912ad-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "a8d760a6-4b8d-462b-b797-75efb61912ad" (UID: "a8d760a6-4b8d-462b-b797-75efb61912ad"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:31:05.199315 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:05.199281    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8d760a6-4b8d-462b-b797-75efb61912ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a8d760a6-4b8d-462b-b797-75efb61912ad" (UID: "a8d760a6-4b8d-462b-b797-75efb61912ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:31:05.199462 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:05.199323    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d760a6-4b8d-462b-b797-75efb61912ad-kube-api-access-cdktn" (OuterVolumeSpecName: "kube-api-access-cdktn") pod "a8d760a6-4b8d-462b-b797-75efb61912ad" (UID: "a8d760a6-4b8d-462b-b797-75efb61912ad"). InnerVolumeSpecName "kube-api-access-cdktn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:31:05.298359 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:05.298201    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8d760a6-4b8d-462b-b797-75efb61912ad-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:31:05.298359 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:05.298233    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdktn\" (UniqueName: \"kubernetes.io/projected/a8d760a6-4b8d-462b-b797-75efb61912ad-kube-api-access-cdktn\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:31:05.298359 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:05.298249    2583 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a8d760a6-4b8d-462b-b797-75efb61912ad-tokenizer-uds\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:31:05.856326 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:05.856290    2583 generic.go:356] "Generic (PLEG): container finished" podID="a8d760a6-4b8d-462b-b797-75efb61912ad" containerID="a7e9d36a6af790e46c609b97a49b967705edb86a2e0f7c972fea5308fbfa9498" exitCode=0
Jul 02 14:31:05.856846 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:05.856340    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j"
Jul 02 14:31:05.856846 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:05.856364    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j" event={"ID":"a8d760a6-4b8d-462b-b797-75efb61912ad","Type":"ContainerDied","Data":"a7e9d36a6af790e46c609b97a49b967705edb86a2e0f7c972fea5308fbfa9498"}
Jul 02 14:31:05.856846 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:05.856434    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j" event={"ID":"a8d760a6-4b8d-462b-b797-75efb61912ad","Type":"ContainerDied","Data":"aeadf6332d7ec1f17629737e3306c8ed1307ca63d12af622a6d3de95e8780e24"}
Jul 02 14:31:05.856846 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:05.856454    2583 scope.go:117] "RemoveContainer" containerID="a7e9d36a6af790e46c609b97a49b967705edb86a2e0f7c972fea5308fbfa9498"
Jul 02 14:31:05.865732 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:05.865710    2583 scope.go:117] "RemoveContainer" containerID="a7e9d36a6af790e46c609b97a49b967705edb86a2e0f7c972fea5308fbfa9498"
Jul 02 14:31:05.866035 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:31:05.866014    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7e9d36a6af790e46c609b97a49b967705edb86a2e0f7c972fea5308fbfa9498\": container with ID starting with a7e9d36a6af790e46c609b97a49b967705edb86a2e0f7c972fea5308fbfa9498 not found: ID does not exist" containerID="a7e9d36a6af790e46c609b97a49b967705edb86a2e0f7c972fea5308fbfa9498"
Jul 02 14:31:05.866107 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:05.866049    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7e9d36a6af790e46c609b97a49b967705edb86a2e0f7c972fea5308fbfa9498"} err="failed to get container status \"a7e9d36a6af790e46c609b97a49b967705edb86a2e0f7c972fea5308fbfa9498\": rpc error: code = NotFound desc = could not find container \"a7e9d36a6af790e46c609b97a49b967705edb86a2e0f7c972fea5308fbfa9498\": container with ID starting with a7e9d36a6af790e46c609b97a49b967705edb86a2e0f7c972fea5308fbfa9498 not found: ID does not exist"
Jul 02 14:31:05.872923 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:05.872894    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j"]
Jul 02 14:31:05.877224 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:05.877202    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedw5v7j"]
Jul 02 14:31:06.338937 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:06.338904    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:31:07.234090 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:07.234056    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8d760a6-4b8d-462b-b797-75efb61912ad" path="/var/lib/kubelet/pods/a8d760a6-4b8d-462b-b797-75efb61912ad/volumes"
Jul 02 14:31:12.532318 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:12.532236    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"
Jul 02 14:31:12.540317 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:12.540287    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"
Jul 02 14:31:19.853448 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:19.853374    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb" podUID="24b229e5-d266-43b2-aaa2-4918404b5063" containerName="main" containerID="cri-o://4d3be208e4425d6be13b2ce1e3d2c522505afda4b35e7183e13bade8a3016a1d" gracePeriod=45
Jul 02 14:31:20.117725 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:20.117696    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb"
Jul 02 14:31:20.239060 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:20.239029    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24b229e5-d266-43b2-aaa2-4918404b5063-tls-certs\") pod \"24b229e5-d266-43b2-aaa2-4918404b5063\" (UID: \"24b229e5-d266-43b2-aaa2-4918404b5063\") "
Jul 02 14:31:20.239060 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:20.239068    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24b229e5-d266-43b2-aaa2-4918404b5063-tmp-dir\") pod \"24b229e5-d266-43b2-aaa2-4918404b5063\" (UID: \"24b229e5-d266-43b2-aaa2-4918404b5063\") "
Jul 02 14:31:20.239325 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:20.239139    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhxjh\" (UniqueName: \"kubernetes.io/projected/24b229e5-d266-43b2-aaa2-4918404b5063-kube-api-access-bhxjh\") pod \"24b229e5-d266-43b2-aaa2-4918404b5063\" (UID: \"24b229e5-d266-43b2-aaa2-4918404b5063\") "
Jul 02 14:31:20.239325 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:20.239170    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24b229e5-d266-43b2-aaa2-4918404b5063-dshm\") pod \"24b229e5-d266-43b2-aaa2-4918404b5063\" (UID: \"24b229e5-d266-43b2-aaa2-4918404b5063\") "
Jul 02 14:31:20.239325 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:20.239202    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24b229e5-d266-43b2-aaa2-4918404b5063-home\") pod \"24b229e5-d266-43b2-aaa2-4918404b5063\" (UID: \"24b229e5-d266-43b2-aaa2-4918404b5063\") "
Jul 02 14:31:20.239325 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:20.239258    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24b229e5-d266-43b2-aaa2-4918404b5063-model-cache\") pod \"24b229e5-d266-43b2-aaa2-4918404b5063\" (UID: \"24b229e5-d266-43b2-aaa2-4918404b5063\") "
Jul 02 14:31:20.239589 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:20.239361    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b229e5-d266-43b2-aaa2-4918404b5063-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "24b229e5-d266-43b2-aaa2-4918404b5063" (UID: "24b229e5-d266-43b2-aaa2-4918404b5063"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:31:20.239589 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:20.239567    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24b229e5-d266-43b2-aaa2-4918404b5063-tmp-dir\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:31:20.239589 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:20.239564    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b229e5-d266-43b2-aaa2-4918404b5063-home" (OuterVolumeSpecName: "home") pod "24b229e5-d266-43b2-aaa2-4918404b5063" (UID: "24b229e5-d266-43b2-aaa2-4918404b5063"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:31:20.239755 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:20.239651    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b229e5-d266-43b2-aaa2-4918404b5063-model-cache" (OuterVolumeSpecName: "model-cache") pod "24b229e5-d266-43b2-aaa2-4918404b5063" (UID: "24b229e5-d266-43b2-aaa2-4918404b5063"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:31:20.241426 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:20.241370    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b229e5-d266-43b2-aaa2-4918404b5063-dshm" (OuterVolumeSpecName: "dshm") pod "24b229e5-d266-43b2-aaa2-4918404b5063" (UID: "24b229e5-d266-43b2-aaa2-4918404b5063"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:31:20.241426 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:20.241384    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b229e5-d266-43b2-aaa2-4918404b5063-kube-api-access-bhxjh" (OuterVolumeSpecName: "kube-api-access-bhxjh") pod "24b229e5-d266-43b2-aaa2-4918404b5063" (UID: "24b229e5-d266-43b2-aaa2-4918404b5063"). InnerVolumeSpecName "kube-api-access-bhxjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:31:20.241426 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:20.241420    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24b229e5-d266-43b2-aaa2-4918404b5063-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "24b229e5-d266-43b2-aaa2-4918404b5063" (UID: "24b229e5-d266-43b2-aaa2-4918404b5063"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:31:20.341092 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:20.341056    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24b229e5-d266-43b2-aaa2-4918404b5063-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:31:20.341092 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:20.341092    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhxjh\" (UniqueName: \"kubernetes.io/projected/24b229e5-d266-43b2-aaa2-4918404b5063-kube-api-access-bhxjh\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:31:20.341377 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:20.341106    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24b229e5-d266-43b2-aaa2-4918404b5063-dshm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:31:20.341377 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:20.341121    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24b229e5-d266-43b2-aaa2-4918404b5063-home\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:31:20.341377 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:20.341133    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24b229e5-d266-43b2-aaa2-4918404b5063-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:31:20.933783 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:20.933749    2583 generic.go:356] "Generic (PLEG): container finished" podID="24b229e5-d266-43b2-aaa2-4918404b5063" containerID="4d3be208e4425d6be13b2ce1e3d2c522505afda4b35e7183e13bade8a3016a1d" exitCode=0
Jul 02 14:31:20.934222 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:20.933803    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb"
Jul 02 14:31:20.934222 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:20.933831    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb" event={"ID":"24b229e5-d266-43b2-aaa2-4918404b5063","Type":"ContainerDied","Data":"4d3be208e4425d6be13b2ce1e3d2c522505afda4b35e7183e13bade8a3016a1d"}
Jul 02 14:31:20.934222 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:20.933892    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb" event={"ID":"24b229e5-d266-43b2-aaa2-4918404b5063","Type":"ContainerDied","Data":"da058fa648c817c4408a60094f72cc0b87623ea2d2d519213a7ee9b86433b9f5"}
Jul 02 14:31:20.934222 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:20.933915    2583 scope.go:117] "RemoveContainer" containerID="4d3be208e4425d6be13b2ce1e3d2c522505afda4b35e7183e13bade8a3016a1d"
Jul 02 14:31:20.943946 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:20.943923    2583 scope.go:117] "RemoveContainer" containerID="4d3be208e4425d6be13b2ce1e3d2c522505afda4b35e7183e13bade8a3016a1d"
Jul 02 14:31:20.944236 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:31:20.944217    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d3be208e4425d6be13b2ce1e3d2c522505afda4b35e7183e13bade8a3016a1d\": container with ID starting with 4d3be208e4425d6be13b2ce1e3d2c522505afda4b35e7183e13bade8a3016a1d not found: ID does not exist" containerID="4d3be208e4425d6be13b2ce1e3d2c522505afda4b35e7183e13bade8a3016a1d"
Jul 02 14:31:20.944282 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:20.944243    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d3be208e4425d6be13b2ce1e3d2c522505afda4b35e7183e13bade8a3016a1d"} err="failed to get container status \"4d3be208e4425d6be13b2ce1e3d2c522505afda4b35e7183e13bade8a3016a1d\": rpc error: code = NotFound desc = could not find container \"4d3be208e4425d6be13b2ce1e3d2c522505afda4b35e7183e13bade8a3016a1d\": container with ID starting with 4d3be208e4425d6be13b2ce1e3d2c522505afda4b35e7183e13bade8a3016a1d not found: ID does not exist"
Jul 02 14:31:20.956930 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:20.956899    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb"]
Jul 02 14:31:20.963467 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:20.963440    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5zwzfb"]
Jul 02 14:31:21.233945 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:21.233853    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24b229e5-d266-43b2-aaa2-4918404b5063" path="/var/lib/kubelet/pods/24b229e5-d266-43b2-aaa2-4918404b5063/volumes"
Jul 02 14:31:31.108572 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:31.108525    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"]
Jul 02 14:31:31.115194 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:31.115163    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz"]
Jul 02 14:31:31.124633 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:31:31.124292    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 02 14:31:31.124633 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 14:31:31.124633 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 14:31:31.124633 ip-10-0-135-143 kubenswrapper[2583]:  > containerID="9dacfb2946dbf34448474803b344ced5f891da56ff7dbf4000b8be6d035c95fc" cmd=["/bin/sleep","15"]
Jul 02 14:31:31.124633 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:31.124364    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz" message="" err=<
Jul 02 14:31:31.124633 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 14:31:31.124633 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 14:31:31.124633 ip-10-0-135-143 kubenswrapper[2583]:  >
Jul 02 14:31:31.124633 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:31:31.124381    2583 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 02 14:31:31.124633 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 14:31:31.124633 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 14:31:31.124633 ip-10-0-135-143 kubenswrapper[2583]:  > pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz" podUID="bce44fe0-3470-4f9d-bf94-487cfb93477d" containerName="main" containerID="cri-o://9dacfb2946dbf34448474803b344ced5f891da56ff7dbf4000b8be6d035c95fc"
Jul 02 14:31:31.124633 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:31.124472    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz" podUID="bce44fe0-3470-4f9d-bf94-487cfb93477d" containerName="main" containerID="cri-o://9dacfb2946dbf34448474803b344ced5f891da56ff7dbf4000b8be6d035c95fc" gracePeriod=60
Jul 02 14:31:31.139134 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:31.139087    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:31:31.205851 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:31.205816    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:31:31.387375 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:31.387347    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz"
Jul 02 14:31:31.561454 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:31.561419    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bce44fe0-3470-4f9d-bf94-487cfb93477d-tls-certs\") pod \"bce44fe0-3470-4f9d-bf94-487cfb93477d\" (UID: \"bce44fe0-3470-4f9d-bf94-487cfb93477d\") "
Jul 02 14:31:31.561658 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:31.561487    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kx9vt\" (UniqueName: \"kubernetes.io/projected/bce44fe0-3470-4f9d-bf94-487cfb93477d-kube-api-access-kx9vt\") pod \"bce44fe0-3470-4f9d-bf94-487cfb93477d\" (UID: \"bce44fe0-3470-4f9d-bf94-487cfb93477d\") "
Jul 02 14:31:31.561658 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:31.561535    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bce44fe0-3470-4f9d-bf94-487cfb93477d-tokenizer-uds\") pod \"bce44fe0-3470-4f9d-bf94-487cfb93477d\" (UID: \"bce44fe0-3470-4f9d-bf94-487cfb93477d\") "
Jul 02 14:31:31.561658 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:31.561563    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bce44fe0-3470-4f9d-bf94-487cfb93477d-kserve-provision-location\") pod \"bce44fe0-3470-4f9d-bf94-487cfb93477d\" (UID: \"bce44fe0-3470-4f9d-bf94-487cfb93477d\") "
Jul 02 14:31:31.561878 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:31.561849    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce44fe0-3470-4f9d-bf94-487cfb93477d-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "bce44fe0-3470-4f9d-bf94-487cfb93477d" (UID: "bce44fe0-3470-4f9d-bf94-487cfb93477d"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:31:31.562265 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:31.562244    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce44fe0-3470-4f9d-bf94-487cfb93477d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bce44fe0-3470-4f9d-bf94-487cfb93477d" (UID: "bce44fe0-3470-4f9d-bf94-487cfb93477d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:31:31.563645 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:31.563616    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bce44fe0-3470-4f9d-bf94-487cfb93477d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bce44fe0-3470-4f9d-bf94-487cfb93477d" (UID: "bce44fe0-3470-4f9d-bf94-487cfb93477d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:31:31.563768 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:31.563718    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bce44fe0-3470-4f9d-bf94-487cfb93477d-kube-api-access-kx9vt" (OuterVolumeSpecName: "kube-api-access-kx9vt") pod "bce44fe0-3470-4f9d-bf94-487cfb93477d" (UID: "bce44fe0-3470-4f9d-bf94-487cfb93477d"). InnerVolumeSpecName "kube-api-access-kx9vt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:31:31.662693 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:31.662652    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bce44fe0-3470-4f9d-bf94-487cfb93477d-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:31:31.662693 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:31.662687    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx9vt\" (UniqueName: \"kubernetes.io/projected/bce44fe0-3470-4f9d-bf94-487cfb93477d-kube-api-access-kx9vt\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:31:31.662693 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:31.662702    2583 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bce44fe0-3470-4f9d-bf94-487cfb93477d-tokenizer-uds\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:31:31.662950 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:31.662713    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bce44fe0-3470-4f9d-bf94-487cfb93477d-kserve-provision-location\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:31:31.991372 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:31.991281    2583 generic.go:356] "Generic (PLEG): container finished" podID="bce44fe0-3470-4f9d-bf94-487cfb93477d" containerID="9dacfb2946dbf34448474803b344ced5f891da56ff7dbf4000b8be6d035c95fc" exitCode=0
Jul 02 14:31:31.991372 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:31.991335    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz"
Jul 02 14:31:31.991372 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:31.991364    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz" event={"ID":"bce44fe0-3470-4f9d-bf94-487cfb93477d","Type":"ContainerDied","Data":"9dacfb2946dbf34448474803b344ced5f891da56ff7dbf4000b8be6d035c95fc"}
Jul 02 14:31:31.991618 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:31.991433    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz" event={"ID":"bce44fe0-3470-4f9d-bf94-487cfb93477d","Type":"ContainerDied","Data":"c564f4185f838c989a677510da514eecea0f0513504edd45da11da9ae5886fbf"}
Jul 02 14:31:31.991618 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:31.991457    2583 scope.go:117] "RemoveContainer" containerID="9dacfb2946dbf34448474803b344ced5f891da56ff7dbf4000b8be6d035c95fc"
Jul 02 14:31:32.000846 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:32.000824    2583 scope.go:117] "RemoveContainer" containerID="345c604bdb58cf2eeee152c70c8e0eee3ebc51636fe60fb05da82f7f6b88259c"
Jul 02 14:31:32.008863 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:32.008844    2583 scope.go:117] "RemoveContainer" containerID="9dacfb2946dbf34448474803b344ced5f891da56ff7dbf4000b8be6d035c95fc"
Jul 02 14:31:32.009145 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:31:32.009123    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dacfb2946dbf34448474803b344ced5f891da56ff7dbf4000b8be6d035c95fc\": container with ID starting with 9dacfb2946dbf34448474803b344ced5f891da56ff7dbf4000b8be6d035c95fc not found: ID does not exist" containerID="9dacfb2946dbf34448474803b344ced5f891da56ff7dbf4000b8be6d035c95fc"
Jul 02 14:31:32.009218 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:32.009158    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dacfb2946dbf34448474803b344ced5f891da56ff7dbf4000b8be6d035c95fc"} err="failed to get container status \"9dacfb2946dbf34448474803b344ced5f891da56ff7dbf4000b8be6d035c95fc\": rpc error: code = NotFound desc = could not find container \"9dacfb2946dbf34448474803b344ced5f891da56ff7dbf4000b8be6d035c95fc\": container with ID starting with 9dacfb2946dbf34448474803b344ced5f891da56ff7dbf4000b8be6d035c95fc not found: ID does not exist"
Jul 02 14:31:32.009218 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:32.009184    2583 scope.go:117] "RemoveContainer" containerID="345c604bdb58cf2eeee152c70c8e0eee3ebc51636fe60fb05da82f7f6b88259c"
Jul 02 14:31:32.009475 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:31:32.009452    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"345c604bdb58cf2eeee152c70c8e0eee3ebc51636fe60fb05da82f7f6b88259c\": container with ID starting with 345c604bdb58cf2eeee152c70c8e0eee3ebc51636fe60fb05da82f7f6b88259c not found: ID does not exist" containerID="345c604bdb58cf2eeee152c70c8e0eee3ebc51636fe60fb05da82f7f6b88259c"
Jul 02 14:31:32.009537 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:32.009481    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"345c604bdb58cf2eeee152c70c8e0eee3ebc51636fe60fb05da82f7f6b88259c"} err="failed to get container status \"345c604bdb58cf2eeee152c70c8e0eee3ebc51636fe60fb05da82f7f6b88259c\": rpc error: code = NotFound desc = could not find container \"345c604bdb58cf2eeee152c70c8e0eee3ebc51636fe60fb05da82f7f6b88259c\": container with ID starting with 345c604bdb58cf2eeee152c70c8e0eee3ebc51636fe60fb05da82f7f6b88259c not found: ID does not exist"
Jul 02 14:31:32.014335 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:32.014310    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz"]
Jul 02 14:31:32.020650 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:32.020627    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-799cb54bc9-pjcfz"]
Jul 02 14:31:33.234441 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:33.234371    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bce44fe0-3470-4f9d-bf94-487cfb93477d" path="/var/lib/kubelet/pods/bce44fe0-3470-4f9d-bf94-487cfb93477d/volumes"
Jul 02 14:31:36.695879 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:36.695847    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:31:43.300658 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.300624    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k"]
Jul 02 14:31:43.301056 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.301026    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bce44fe0-3470-4f9d-bf94-487cfb93477d" containerName="storage-initializer"
Jul 02 14:31:43.301056 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.301039    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bce44fe0-3470-4f9d-bf94-487cfb93477d" containerName="storage-initializer"
Jul 02 14:31:43.301056 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.301048    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bce44fe0-3470-4f9d-bf94-487cfb93477d" containerName="main"
Jul 02 14:31:43.301056 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.301056    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bce44fe0-3470-4f9d-bf94-487cfb93477d" containerName="main"
Jul 02 14:31:43.301215 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.301067    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8d760a6-4b8d-462b-b797-75efb61912ad" containerName="main"
Jul 02 14:31:43.301215 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.301073    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8d760a6-4b8d-462b-b797-75efb61912ad" containerName="main"
Jul 02 14:31:43.301215 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.301083    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24b229e5-d266-43b2-aaa2-4918404b5063" containerName="main"
Jul 02 14:31:43.301215 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.301088    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="24b229e5-d266-43b2-aaa2-4918404b5063" containerName="main"
Jul 02 14:31:43.301215 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.301138    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8d760a6-4b8d-462b-b797-75efb61912ad" containerName="main"
Jul 02 14:31:43.301215 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.301145    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="24b229e5-d266-43b2-aaa2-4918404b5063" containerName="main"
Jul 02 14:31:43.301215 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.301156    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bce44fe0-3470-4f9d-bf94-487cfb93477d" containerName="main"
Jul 02 14:31:43.304184 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.304163    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k"
Jul 02 14:31:43.306781 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.306759    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 02 14:31:43.314049 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.314022    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k"]
Jul 02 14:31:43.321132 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.321100    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:31:43.370534 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.370497    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-tls-certs\") pod \"rolling-upgrade-test-kserve-54bc76f55b-nlw4k\" (UID: \"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k"
Jul 02 14:31:43.370534 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.370534    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-dshm\") pod \"rolling-upgrade-test-kserve-54bc76f55b-nlw4k\" (UID: \"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k"
Jul 02 14:31:43.370786 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.370567    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhqd8\" (UniqueName: \"kubernetes.io/projected/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-kube-api-access-bhqd8\") pod \"rolling-upgrade-test-kserve-54bc76f55b-nlw4k\" (UID: \"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k"
Jul 02 14:31:43.370786 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.370606    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-home\") pod \"rolling-upgrade-test-kserve-54bc76f55b-nlw4k\" (UID: \"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k"
Jul 02 14:31:43.370786 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.370671    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-model-cache\") pod \"rolling-upgrade-test-kserve-54bc76f55b-nlw4k\" (UID: \"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k"
Jul 02 14:31:43.370786 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.370757    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-tmp-dir\") pod \"rolling-upgrade-test-kserve-54bc76f55b-nlw4k\" (UID: \"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k"
Jul 02 14:31:43.471985 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.471950    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-tmp-dir\") pod \"rolling-upgrade-test-kserve-54bc76f55b-nlw4k\" (UID: \"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k"
Jul 02 14:31:43.472183 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.472008    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-tls-certs\") pod \"rolling-upgrade-test-kserve-54bc76f55b-nlw4k\" (UID: \"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k"
Jul 02 14:31:43.472183 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.472029    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-dshm\") pod \"rolling-upgrade-test-kserve-54bc76f55b-nlw4k\" (UID: \"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k"
Jul 02 14:31:43.472183 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.472050    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhqd8\" (UniqueName: \"kubernetes.io/projected/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-kube-api-access-bhqd8\") pod \"rolling-upgrade-test-kserve-54bc76f55b-nlw4k\" (UID: \"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k"
Jul 02 14:31:43.472183 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.472076    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-home\") pod \"rolling-upgrade-test-kserve-54bc76f55b-nlw4k\" (UID: \"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k"
Jul 02 14:31:43.472183 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.472118    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-model-cache\") pod \"rolling-upgrade-test-kserve-54bc76f55b-nlw4k\" (UID: \"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k"
Jul 02 14:31:43.472496 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.472361    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-tmp-dir\") pod \"rolling-upgrade-test-kserve-54bc76f55b-nlw4k\" (UID: \"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k"
Jul 02 14:31:43.472752 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.472729    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-home\") pod \"rolling-upgrade-test-kserve-54bc76f55b-nlw4k\" (UID: \"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k"
Jul 02 14:31:43.472832 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.472795    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-model-cache\") pod \"rolling-upgrade-test-kserve-54bc76f55b-nlw4k\" (UID: \"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k"
Jul 02 14:31:43.474857 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.474822    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-dshm\") pod \"rolling-upgrade-test-kserve-54bc76f55b-nlw4k\" (UID: \"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k"
Jul 02 14:31:43.475443 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.475422    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-tls-certs\") pod \"rolling-upgrade-test-kserve-54bc76f55b-nlw4k\" (UID: \"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k"
Jul 02 14:31:43.480693 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.480669    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhqd8\" (UniqueName: \"kubernetes.io/projected/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-kube-api-access-bhqd8\") pod \"rolling-upgrade-test-kserve-54bc76f55b-nlw4k\" (UID: \"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k"
Jul 02 14:31:43.601228 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.601125    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd"]
Jul 02 14:31:43.608327 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.608296    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd"
Jul 02 14:31:43.610833 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.610804    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"rolling-upgrade-test-epp-sa-dockercfg-vz9sk\""
Jul 02 14:31:43.615265 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.615240    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k"
Jul 02 14:31:43.620183 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.620156    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd"]
Jul 02 14:31:43.640891 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.640857    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:31:43.674283 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.674205    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6x42\" (UniqueName: \"kubernetes.io/projected/3f4e645b-25e2-4635-af7f-ff1a6b932402-kube-api-access-n6x42\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd\" (UID: \"3f4e645b-25e2-4635-af7f-ff1a6b932402\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd"
Jul 02 14:31:43.674283 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.674263    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f4e645b-25e2-4635-af7f-ff1a6b932402-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd\" (UID: \"3f4e645b-25e2-4635-af7f-ff1a6b932402\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd"
Jul 02 14:31:43.674561 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.674443    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3f4e645b-25e2-4635-af7f-ff1a6b932402-tokenizer-uds\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd\" (UID: \"3f4e645b-25e2-4635-af7f-ff1a6b932402\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd"
Jul 02 14:31:43.753717 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.753688    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k"]
Jul 02 14:31:43.756162 ip-10-0-135-143 kubenswrapper[2583]: W0702 14:31:43.756130    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dc36c4c_ffb6_4dd4_a6c9_9b75ce3c1ffd.slice/crio-d8b7aa3b9b977306acd14698eb46f196a063e06e999295c25e791f6f26323d71 WatchSource:0}: Error finding container d8b7aa3b9b977306acd14698eb46f196a063e06e999295c25e791f6f26323d71: Status 404 returned error can't find the container with id d8b7aa3b9b977306acd14698eb46f196a063e06e999295c25e791f6f26323d71
Jul 02 14:31:43.776135 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.775958    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3f4e645b-25e2-4635-af7f-ff1a6b932402-tokenizer-uds\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd\" (UID: \"3f4e645b-25e2-4635-af7f-ff1a6b932402\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd"
Jul 02 14:31:43.776135 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.776075    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6x42\" (UniqueName: \"kubernetes.io/projected/3f4e645b-25e2-4635-af7f-ff1a6b932402-kube-api-access-n6x42\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd\" (UID: \"3f4e645b-25e2-4635-af7f-ff1a6b932402\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd"
Jul 02 14:31:43.776135 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.776108    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f4e645b-25e2-4635-af7f-ff1a6b932402-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd\" (UID: \"3f4e645b-25e2-4635-af7f-ff1a6b932402\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd"
Jul 02 14:31:43.776328 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.776308    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3f4e645b-25e2-4635-af7f-ff1a6b932402-tokenizer-uds\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd\" (UID: \"3f4e645b-25e2-4635-af7f-ff1a6b932402\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd"
Jul 02 14:31:43.778748 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.778725    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f4e645b-25e2-4635-af7f-ff1a6b932402-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd\" (UID: \"3f4e645b-25e2-4635-af7f-ff1a6b932402\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd"
Jul 02 14:31:43.784757 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.784735    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6x42\" (UniqueName: \"kubernetes.io/projected/3f4e645b-25e2-4635-af7f-ff1a6b932402-kube-api-access-n6x42\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd\" (UID: \"3f4e645b-25e2-4635-af7f-ff1a6b932402\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd"
Jul 02 14:31:43.921198 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:43.921156    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd"
Jul 02 14:31:44.052051 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:44.052015    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k" event={"ID":"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd","Type":"ContainerStarted","Data":"f5838eb4a59d76112810fc1b53b5dc18e1fdd8cd903f4e3d99c834479831f2a4"}
Jul 02 14:31:44.052230 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:44.052081    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k" event={"ID":"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd","Type":"ContainerStarted","Data":"d8b7aa3b9b977306acd14698eb46f196a063e06e999295c25e791f6f26323d71"}
Jul 02 14:31:44.062994 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:44.062968    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd"]
Jul 02 14:31:44.064824 ip-10-0-135-143 kubenswrapper[2583]: W0702 14:31:44.064796    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f4e645b_25e2_4635_af7f_ff1a6b932402.slice/crio-39c260191573abbcd99195109f47e234edde6122db165aa2293e33dd37113b56 WatchSource:0}: Error finding container 39c260191573abbcd99195109f47e234edde6122db165aa2293e33dd37113b56: Status 404 returned error can't find the container with id 39c260191573abbcd99195109f47e234edde6122db165aa2293e33dd37113b56
Jul 02 14:31:44.069769 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:44.069727    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k" podStartSLOduration=1.069708197 podStartE2EDuration="1.069708197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:31:44.068996985 +0000 UTC m=+4615.296908160" watchObservedRunningTime="2026-07-02 14:31:44.069708197 +0000 UTC m=+4615.297619371"
Jul 02 14:31:44.616144 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:44.616108    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k"
Jul 02 14:31:45.060760 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:45.060665    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd" event={"ID":"3f4e645b-25e2-4635-af7f-ff1a6b932402","Type":"ContainerStarted","Data":"11453d1aa4a34e3c77fc7639450bd17930b6d3f59b593153682a25ffddf46998"}
Jul 02 14:31:45.060760 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:45.060718    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd" event={"ID":"3f4e645b-25e2-4635-af7f-ff1a6b932402","Type":"ContainerStarted","Data":"39c260191573abbcd99195109f47e234edde6122db165aa2293e33dd37113b56"}
Jul 02 14:31:45.060946 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:45.060779    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd"
Jul 02 14:31:45.078576 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:45.078526    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd" podStartSLOduration=2.078508966 podStartE2EDuration="2.078508966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:31:45.07767007 +0000 UTC m=+4616.305581244" watchObservedRunningTime="2026-07-02 14:31:45.078508966 +0000 UTC m=+4616.306420162"
Jul 02 14:31:46.142229 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:46.142183    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx" podUID="54604b57-049c-4ed0-bdd3-cb8a1499218a" containerName="main" containerID="cri-o://5b52970088ff888f0837fceefcaaa1b39c255b028ff2b2b2222850e8dc1ce324" gracePeriod=45
Jul 02 14:31:46.533403 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:46.533281    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx" podUID="54604b57-049c-4ed0-bdd3-cb8a1499218a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Jul 02 14:31:46.895235 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:46.895208    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"
Jul 02 14:31:46.905799 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:46.905765    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54604b57-049c-4ed0-bdd3-cb8a1499218a-tmp-dir\") pod \"54604b57-049c-4ed0-bdd3-cb8a1499218a\" (UID: \"54604b57-049c-4ed0-bdd3-cb8a1499218a\") "
Jul 02 14:31:46.905982 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:46.905828    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54604b57-049c-4ed0-bdd3-cb8a1499218a-model-cache\") pod \"54604b57-049c-4ed0-bdd3-cb8a1499218a\" (UID: \"54604b57-049c-4ed0-bdd3-cb8a1499218a\") "
Jul 02 14:31:46.905982 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:46.905881    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54604b57-049c-4ed0-bdd3-cb8a1499218a-home\") pod \"54604b57-049c-4ed0-bdd3-cb8a1499218a\" (UID: \"54604b57-049c-4ed0-bdd3-cb8a1499218a\") "
Jul 02 14:31:46.905982 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:46.905906    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54604b57-049c-4ed0-bdd3-cb8a1499218a-tls-certs\") pod \"54604b57-049c-4ed0-bdd3-cb8a1499218a\" (UID: \"54604b57-049c-4ed0-bdd3-cb8a1499218a\") "
Jul 02 14:31:46.905982 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:46.905932    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54604b57-049c-4ed0-bdd3-cb8a1499218a-kserve-provision-location\") pod \"54604b57-049c-4ed0-bdd3-cb8a1499218a\" (UID: \"54604b57-049c-4ed0-bdd3-cb8a1499218a\") "
Jul 02 14:31:46.905982 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:46.905959    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54604b57-049c-4ed0-bdd3-cb8a1499218a-dshm\") pod \"54604b57-049c-4ed0-bdd3-cb8a1499218a\" (UID: \"54604b57-049c-4ed0-bdd3-cb8a1499218a\") "
Jul 02 14:31:46.906340 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:46.906003    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzlkw\" (UniqueName: \"kubernetes.io/projected/54604b57-049c-4ed0-bdd3-cb8a1499218a-kube-api-access-bzlkw\") pod \"54604b57-049c-4ed0-bdd3-cb8a1499218a\" (UID: \"54604b57-049c-4ed0-bdd3-cb8a1499218a\") "
Jul 02 14:31:46.906340 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:46.906201    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54604b57-049c-4ed0-bdd3-cb8a1499218a-model-cache" (OuterVolumeSpecName: "model-cache") pod "54604b57-049c-4ed0-bdd3-cb8a1499218a" (UID: "54604b57-049c-4ed0-bdd3-cb8a1499218a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:31:46.906495 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:46.906414    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54604b57-049c-4ed0-bdd3-cb8a1499218a-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:31:46.906587 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:46.906562    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54604b57-049c-4ed0-bdd3-cb8a1499218a-home" (OuterVolumeSpecName: "home") pod "54604b57-049c-4ed0-bdd3-cb8a1499218a" (UID: "54604b57-049c-4ed0-bdd3-cb8a1499218a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:31:46.908170 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:46.908147    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54604b57-049c-4ed0-bdd3-cb8a1499218a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "54604b57-049c-4ed0-bdd3-cb8a1499218a" (UID: "54604b57-049c-4ed0-bdd3-cb8a1499218a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:31:46.908737 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:46.908718    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54604b57-049c-4ed0-bdd3-cb8a1499218a-dshm" (OuterVolumeSpecName: "dshm") pod "54604b57-049c-4ed0-bdd3-cb8a1499218a" (UID: "54604b57-049c-4ed0-bdd3-cb8a1499218a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:31:46.908828 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:46.908745    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54604b57-049c-4ed0-bdd3-cb8a1499218a-kube-api-access-bzlkw" (OuterVolumeSpecName: "kube-api-access-bzlkw") pod "54604b57-049c-4ed0-bdd3-cb8a1499218a" (UID: "54604b57-049c-4ed0-bdd3-cb8a1499218a"). InnerVolumeSpecName "kube-api-access-bzlkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:31:46.917679 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:46.917591    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54604b57-049c-4ed0-bdd3-cb8a1499218a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "54604b57-049c-4ed0-bdd3-cb8a1499218a" (UID: "54604b57-049c-4ed0-bdd3-cb8a1499218a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:31:46.983405 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:46.983358    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54604b57-049c-4ed0-bdd3-cb8a1499218a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "54604b57-049c-4ed0-bdd3-cb8a1499218a" (UID: "54604b57-049c-4ed0-bdd3-cb8a1499218a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:31:47.007241 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:47.007210    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54604b57-049c-4ed0-bdd3-cb8a1499218a-home\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:31:47.007241 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:47.007241    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54604b57-049c-4ed0-bdd3-cb8a1499218a-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:31:47.007411 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:47.007253    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54604b57-049c-4ed0-bdd3-cb8a1499218a-kserve-provision-location\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:31:47.007411 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:47.007261    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54604b57-049c-4ed0-bdd3-cb8a1499218a-dshm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:31:47.007411 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:47.007285    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzlkw\" (UniqueName: \"kubernetes.io/projected/54604b57-049c-4ed0-bdd3-cb8a1499218a-kube-api-access-bzlkw\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:31:47.007411 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:47.007295    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54604b57-049c-4ed0-bdd3-cb8a1499218a-tmp-dir\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:31:47.073479 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:47.073379    2583 generic.go:356] "Generic (PLEG): container finished" podID="54604b57-049c-4ed0-bdd3-cb8a1499218a" containerID="5b52970088ff888f0837fceefcaaa1b39c255b028ff2b2b2222850e8dc1ce324" exitCode=0
Jul 02 14:31:47.073479 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:47.073419    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx" event={"ID":"54604b57-049c-4ed0-bdd3-cb8a1499218a","Type":"ContainerDied","Data":"5b52970088ff888f0837fceefcaaa1b39c255b028ff2b2b2222850e8dc1ce324"}
Jul 02 14:31:47.073479 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:47.073465    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"
Jul 02 14:31:47.073479 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:47.073474    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx" event={"ID":"54604b57-049c-4ed0-bdd3-cb8a1499218a","Type":"ContainerDied","Data":"40765db642f8c7886f43e6307cb976d285474903316426bd4dcaf784e3caa9cc"}
Jul 02 14:31:47.073746 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:47.073497    2583 scope.go:117] "RemoveContainer" containerID="5b52970088ff888f0837fceefcaaa1b39c255b028ff2b2b2222850e8dc1ce324"
Jul 02 14:31:47.083269 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:47.083252    2583 scope.go:117] "RemoveContainer" containerID="86ef3ae669e7d6af274813ef1a84480944519423f8dba3fc49cdf69ce57eef97"
Jul 02 14:31:47.091811 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:47.091791    2583 scope.go:117] "RemoveContainer" containerID="5b52970088ff888f0837fceefcaaa1b39c255b028ff2b2b2222850e8dc1ce324"
Jul 02 14:31:47.092106 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:31:47.092086    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b52970088ff888f0837fceefcaaa1b39c255b028ff2b2b2222850e8dc1ce324\": container with ID starting with 5b52970088ff888f0837fceefcaaa1b39c255b028ff2b2b2222850e8dc1ce324 not found: ID does not exist" containerID="5b52970088ff888f0837fceefcaaa1b39c255b028ff2b2b2222850e8dc1ce324"
Jul 02 14:31:47.092159 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:47.092120    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b52970088ff888f0837fceefcaaa1b39c255b028ff2b2b2222850e8dc1ce324"} err="failed to get container status \"5b52970088ff888f0837fceefcaaa1b39c255b028ff2b2b2222850e8dc1ce324\": rpc error: code = NotFound desc = could not find container \"5b52970088ff888f0837fceefcaaa1b39c255b028ff2b2b2222850e8dc1ce324\": container with ID starting with 5b52970088ff888f0837fceefcaaa1b39c255b028ff2b2b2222850e8dc1ce324 not found: ID does not exist"
Jul 02 14:31:47.092159 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:47.092140    2583 scope.go:117] "RemoveContainer" containerID="86ef3ae669e7d6af274813ef1a84480944519423f8dba3fc49cdf69ce57eef97"
Jul 02 14:31:47.092460 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:31:47.092434    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86ef3ae669e7d6af274813ef1a84480944519423f8dba3fc49cdf69ce57eef97\": container with ID starting with 86ef3ae669e7d6af274813ef1a84480944519423f8dba3fc49cdf69ce57eef97 not found: ID does not exist" containerID="86ef3ae669e7d6af274813ef1a84480944519423f8dba3fc49cdf69ce57eef97"
Jul 02 14:31:47.092537 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:47.092468    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86ef3ae669e7d6af274813ef1a84480944519423f8dba3fc49cdf69ce57eef97"} err="failed to get container status \"86ef3ae669e7d6af274813ef1a84480944519423f8dba3fc49cdf69ce57eef97\": rpc error: code = NotFound desc = could not find container \"86ef3ae669e7d6af274813ef1a84480944519423f8dba3fc49cdf69ce57eef97\": container with ID starting with 86ef3ae669e7d6af274813ef1a84480944519423f8dba3fc49cdf69ce57eef97 not found: ID does not exist"
Jul 02 14:31:47.099097 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:47.099068    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"]
Jul 02 14:31:47.103930 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:47.103903    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-cbc87bf9d-g9hvx"]
Jul 02 14:31:47.235607 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:47.235574    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54604b57-049c-4ed0-bdd3-cb8a1499218a" path="/var/lib/kubelet/pods/54604b57-049c-4ed0-bdd3-cb8a1499218a/volumes"
Jul 02 14:31:53.615595 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:53.615558    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k"
Jul 02 14:31:53.628226 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:53.628197    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k"
Jul 02 14:31:54.118548 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:54.118520    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k"
Jul 02 14:31:54.144175 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:31:54.144142    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:32:05.493321 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:05.493283    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:32:11.715932 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:11.715891    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:32:16.067821 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:16.067789    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd"
Jul 02 14:32:26.916023 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:26.915981    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:32:27.016623 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:27.016583    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:32:35.708284 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:35.708251    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"]
Jul 02 14:32:35.708787 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:35.708637    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54604b57-049c-4ed0-bdd3-cb8a1499218a" containerName="storage-initializer"
Jul 02 14:32:35.708787 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:35.708648    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="54604b57-049c-4ed0-bdd3-cb8a1499218a" containerName="storage-initializer"
Jul 02 14:32:35.708787 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:35.708659    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54604b57-049c-4ed0-bdd3-cb8a1499218a" containerName="main"
Jul 02 14:32:35.708787 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:35.708665    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="54604b57-049c-4ed0-bdd3-cb8a1499218a" containerName="main"
Jul 02 14:32:35.708787 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:35.708734    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="54604b57-049c-4ed0-bdd3-cb8a1499218a" containerName="main"
Jul 02 14:32:35.712222 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:35.712200    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"
Jul 02 14:32:35.714938 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:35.714914    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 14:32:35.715061 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:35.714914    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 02 14:32:35.723780 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:35.723751    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"]
Jul 02 14:32:35.784940 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:35.784900    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f52dac5-8345-47b7-9f79-b538910f1d32-model-cache\") pod \"prestop-hook-test-kserve-c4c794bb6-dts94\" (UID: \"2f52dac5-8345-47b7-9f79-b538910f1d32\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"
Jul 02 14:32:35.785116 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:35.784984    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f52dac5-8345-47b7-9f79-b538910f1d32-kserve-provision-location\") pod \"prestop-hook-test-kserve-c4c794bb6-dts94\" (UID: \"2f52dac5-8345-47b7-9f79-b538910f1d32\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"
Jul 02 14:32:35.785116 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:35.785024    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f52dac5-8345-47b7-9f79-b538910f1d32-home\") pod \"prestop-hook-test-kserve-c4c794bb6-dts94\" (UID: \"2f52dac5-8345-47b7-9f79-b538910f1d32\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"
Jul 02 14:32:35.785116 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:35.785107    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f52dac5-8345-47b7-9f79-b538910f1d32-tmp-dir\") pod \"prestop-hook-test-kserve-c4c794bb6-dts94\" (UID: \"2f52dac5-8345-47b7-9f79-b538910f1d32\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"
Jul 02 14:32:35.785244 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:35.785228    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f52dac5-8345-47b7-9f79-b538910f1d32-dshm\") pod \"prestop-hook-test-kserve-c4c794bb6-dts94\" (UID: \"2f52dac5-8345-47b7-9f79-b538910f1d32\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"
Jul 02 14:32:35.785290 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:35.785253    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d86ds\" (UniqueName: \"kubernetes.io/projected/2f52dac5-8345-47b7-9f79-b538910f1d32-kube-api-access-d86ds\") pod \"prestop-hook-test-kserve-c4c794bb6-dts94\" (UID: \"2f52dac5-8345-47b7-9f79-b538910f1d32\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"
Jul 02 14:32:35.785290 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:35.785274    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f52dac5-8345-47b7-9f79-b538910f1d32-tls-certs\") pod \"prestop-hook-test-kserve-c4c794bb6-dts94\" (UID: \"2f52dac5-8345-47b7-9f79-b538910f1d32\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"
Jul 02 14:32:35.791338 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:35.791311    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:32:35.886535 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:35.886495    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f52dac5-8345-47b7-9f79-b538910f1d32-dshm\") pod \"prestop-hook-test-kserve-c4c794bb6-dts94\" (UID: \"2f52dac5-8345-47b7-9f79-b538910f1d32\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"
Jul 02 14:32:35.886535 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:35.886538    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d86ds\" (UniqueName: \"kubernetes.io/projected/2f52dac5-8345-47b7-9f79-b538910f1d32-kube-api-access-d86ds\") pod \"prestop-hook-test-kserve-c4c794bb6-dts94\" (UID: \"2f52dac5-8345-47b7-9f79-b538910f1d32\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"
Jul 02 14:32:35.886778 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:35.886560    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f52dac5-8345-47b7-9f79-b538910f1d32-tls-certs\") pod \"prestop-hook-test-kserve-c4c794bb6-dts94\" (UID: \"2f52dac5-8345-47b7-9f79-b538910f1d32\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"
Jul 02 14:32:35.886778 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:35.886595    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f52dac5-8345-47b7-9f79-b538910f1d32-model-cache\") pod \"prestop-hook-test-kserve-c4c794bb6-dts94\" (UID: \"2f52dac5-8345-47b7-9f79-b538910f1d32\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"
Jul 02 14:32:35.886778 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:35.886624    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f52dac5-8345-47b7-9f79-b538910f1d32-kserve-provision-location\") pod \"prestop-hook-test-kserve-c4c794bb6-dts94\" (UID: \"2f52dac5-8345-47b7-9f79-b538910f1d32\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"
Jul 02 14:32:35.886778 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:35.886650    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f52dac5-8345-47b7-9f79-b538910f1d32-home\") pod \"prestop-hook-test-kserve-c4c794bb6-dts94\" (UID: \"2f52dac5-8345-47b7-9f79-b538910f1d32\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"
Jul 02 14:32:35.886778 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:35.886707    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f52dac5-8345-47b7-9f79-b538910f1d32-tmp-dir\") pod \"prestop-hook-test-kserve-c4c794bb6-dts94\" (UID: \"2f52dac5-8345-47b7-9f79-b538910f1d32\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"
Jul 02 14:32:35.887067 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:35.887040    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f52dac5-8345-47b7-9f79-b538910f1d32-model-cache\") pod \"prestop-hook-test-kserve-c4c794bb6-dts94\" (UID: \"2f52dac5-8345-47b7-9f79-b538910f1d32\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"
Jul 02 14:32:35.887134 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:35.887080    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f52dac5-8345-47b7-9f79-b538910f1d32-tmp-dir\") pod \"prestop-hook-test-kserve-c4c794bb6-dts94\" (UID: \"2f52dac5-8345-47b7-9f79-b538910f1d32\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"
Jul 02 14:32:35.887134 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:35.887114    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f52dac5-8345-47b7-9f79-b538910f1d32-kserve-provision-location\") pod \"prestop-hook-test-kserve-c4c794bb6-dts94\" (UID: \"2f52dac5-8345-47b7-9f79-b538910f1d32\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"
Jul 02 14:32:35.887234 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:35.887167    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f52dac5-8345-47b7-9f79-b538910f1d32-home\") pod \"prestop-hook-test-kserve-c4c794bb6-dts94\" (UID: \"2f52dac5-8345-47b7-9f79-b538910f1d32\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"
Jul 02 14:32:35.888882 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:35.888855    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f52dac5-8345-47b7-9f79-b538910f1d32-dshm\") pod \"prestop-hook-test-kserve-c4c794bb6-dts94\" (UID: \"2f52dac5-8345-47b7-9f79-b538910f1d32\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"
Jul 02 14:32:35.889149 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:35.889128    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f52dac5-8345-47b7-9f79-b538910f1d32-tls-certs\") pod \"prestop-hook-test-kserve-c4c794bb6-dts94\" (UID: \"2f52dac5-8345-47b7-9f79-b538910f1d32\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"
Jul 02 14:32:35.895157 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:35.895132    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d86ds\" (UniqueName: \"kubernetes.io/projected/2f52dac5-8345-47b7-9f79-b538910f1d32-kube-api-access-d86ds\") pod \"prestop-hook-test-kserve-c4c794bb6-dts94\" (UID: \"2f52dac5-8345-47b7-9f79-b538910f1d32\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"
Jul 02 14:32:36.023358 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:36.023261    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"
Jul 02 14:32:36.168612 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:36.168584    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"]
Jul 02 14:32:36.170720 ip-10-0-135-143 kubenswrapper[2583]: W0702 14:32:36.170682    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f52dac5_8345_47b7_9f79_b538910f1d32.slice/crio-c9eb739919d14326627a4b8e1a848ff9de6e1be6b592ce46051674807ea2e034 WatchSource:0}: Error finding container c9eb739919d14326627a4b8e1a848ff9de6e1be6b592ce46051674807ea2e034: Status 404 returned error can't find the container with id c9eb739919d14326627a4b8e1a848ff9de6e1be6b592ce46051674807ea2e034
Jul 02 14:32:36.215922 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:36.215892    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx"]
Jul 02 14:32:36.221470 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:36.221445    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx"
Jul 02 14:32:36.224783 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:36.224762    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-epp-sa-dockercfg-r7m6x\""
Jul 02 14:32:36.236524 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:36.236493    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx"]
Jul 02 14:32:36.259405 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:36.259348    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:32:36.290854 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:36.290753    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j5rh\" (UniqueName: \"kubernetes.io/projected/65cf02af-a227-41ed-9445-9de19fad5cc9-kube-api-access-2j5rh\") pod \"prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx\" (UID: \"65cf02af-a227-41ed-9445-9de19fad5cc9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx"
Jul 02 14:32:36.290854 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:36.290831    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65cf02af-a227-41ed-9445-9de19fad5cc9-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx\" (UID: \"65cf02af-a227-41ed-9445-9de19fad5cc9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx"
Jul 02 14:32:36.291115 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:36.290881    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/65cf02af-a227-41ed-9445-9de19fad5cc9-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx\" (UID: \"65cf02af-a227-41ed-9445-9de19fad5cc9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx"
Jul 02 14:32:36.291115 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:36.290929    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65cf02af-a227-41ed-9445-9de19fad5cc9-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx\" (UID: \"65cf02af-a227-41ed-9445-9de19fad5cc9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx"
Jul 02 14:32:36.312728 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:36.312688    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94" event={"ID":"2f52dac5-8345-47b7-9f79-b538910f1d32","Type":"ContainerStarted","Data":"c7cc18336120a246b1b28f55e41f95b393a2c632d974fb3caa0be5fcb9f805fa"}
Jul 02 14:32:36.312915 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:36.312741    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94" event={"ID":"2f52dac5-8345-47b7-9f79-b538910f1d32","Type":"ContainerStarted","Data":"c9eb739919d14326627a4b8e1a848ff9de6e1be6b592ce46051674807ea2e034"}
Jul 02 14:32:36.392232 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:36.392189    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j5rh\" (UniqueName: \"kubernetes.io/projected/65cf02af-a227-41ed-9445-9de19fad5cc9-kube-api-access-2j5rh\") pod \"prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx\" (UID: \"65cf02af-a227-41ed-9445-9de19fad5cc9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx"
Jul 02 14:32:36.392435 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:36.392301    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65cf02af-a227-41ed-9445-9de19fad5cc9-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx\" (UID: \"65cf02af-a227-41ed-9445-9de19fad5cc9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx"
Jul 02 14:32:36.392528 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:36.392477    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/65cf02af-a227-41ed-9445-9de19fad5cc9-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx\" (UID: \"65cf02af-a227-41ed-9445-9de19fad5cc9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx"
Jul 02 14:32:36.392588 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:36.392566    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65cf02af-a227-41ed-9445-9de19fad5cc9-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx\" (UID: \"65cf02af-a227-41ed-9445-9de19fad5cc9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx"
Jul 02 14:32:36.392899 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:36.392872    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/65cf02af-a227-41ed-9445-9de19fad5cc9-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx\" (UID: \"65cf02af-a227-41ed-9445-9de19fad5cc9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx"
Jul 02 14:32:36.393009 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:36.392960    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65cf02af-a227-41ed-9445-9de19fad5cc9-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx\" (UID: \"65cf02af-a227-41ed-9445-9de19fad5cc9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx"
Jul 02 14:32:36.394970 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:36.394945    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65cf02af-a227-41ed-9445-9de19fad5cc9-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx\" (UID: \"65cf02af-a227-41ed-9445-9de19fad5cc9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx"
Jul 02 14:32:36.402144 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:36.402119    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j5rh\" (UniqueName: \"kubernetes.io/projected/65cf02af-a227-41ed-9445-9de19fad5cc9-kube-api-access-2j5rh\") pod \"prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx\" (UID: \"65cf02af-a227-41ed-9445-9de19fad5cc9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx"
Jul 02 14:32:36.531888 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:36.531848    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx"
Jul 02 14:32:36.673335 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:36.673302    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx"]
Jul 02 14:32:36.675715 ip-10-0-135-143 kubenswrapper[2583]: W0702 14:32:36.675689    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65cf02af_a227_41ed_9445_9de19fad5cc9.slice/crio-3dc7738dfbd820972bea04f498d94a92e2c9ef1356e745b40962dcafd144686d WatchSource:0}: Error finding container 3dc7738dfbd820972bea04f498d94a92e2c9ef1356e745b40962dcafd144686d: Status 404 returned error can't find the container with id 3dc7738dfbd820972bea04f498d94a92e2c9ef1356e745b40962dcafd144686d
Jul 02 14:32:37.323078 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:37.323007    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx" event={"ID":"65cf02af-a227-41ed-9445-9de19fad5cc9","Type":"ContainerStarted","Data":"92c6cea72dc420c9c31e62b66e151be53bd2cf81eed5a2ca526a1bdbc18f04b6"}
Jul 02 14:32:37.323078 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:37.323086    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx" event={"ID":"65cf02af-a227-41ed-9445-9de19fad5cc9","Type":"ContainerStarted","Data":"3dc7738dfbd820972bea04f498d94a92e2c9ef1356e745b40962dcafd144686d"}
Jul 02 14:32:38.330597 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:38.330561    2583 generic.go:356] "Generic (PLEG): container finished" podID="65cf02af-a227-41ed-9445-9de19fad5cc9" containerID="92c6cea72dc420c9c31e62b66e151be53bd2cf81eed5a2ca526a1bdbc18f04b6" exitCode=0
Jul 02 14:32:38.331034 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:38.330639    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx" event={"ID":"65cf02af-a227-41ed-9445-9de19fad5cc9","Type":"ContainerDied","Data":"92c6cea72dc420c9c31e62b66e151be53bd2cf81eed5a2ca526a1bdbc18f04b6"}
Jul 02 14:32:39.338888 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:39.338775    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx" event={"ID":"65cf02af-a227-41ed-9445-9de19fad5cc9","Type":"ContainerStarted","Data":"fbe904c249c46a199cca2a62c102cf302cce39c9e3e72e9cae1dcd7fc30b1ee9"}
Jul 02 14:32:39.338888 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:39.338853    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx"
Jul 02 14:32:39.370238 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:39.370158    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx" podStartSLOduration=2.616400217 podStartE2EDuration="3.370134832s" totalImagesPullingTime="0s" totalInitContainerRuntime="753.734615ms" isStatefulPod=false podCreationTimestamp="2026-07-02 14:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:32:39.369025715 +0000 UTC m=+4670.596936890" watchObservedRunningTime="2026-07-02 14:32:39.370134832 +0000 UTC m=+4670.598046006"
Jul 02 14:32:40.345731 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:40.345697    2583 generic.go:356] "Generic (PLEG): container finished" podID="2f52dac5-8345-47b7-9f79-b538910f1d32" containerID="c7cc18336120a246b1b28f55e41f95b393a2c632d974fb3caa0be5fcb9f805fa" exitCode=0
Jul 02 14:32:40.346114 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:40.345782    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94" event={"ID":"2f52dac5-8345-47b7-9f79-b538910f1d32","Type":"ContainerDied","Data":"c7cc18336120a246b1b28f55e41f95b393a2c632d974fb3caa0be5fcb9f805fa"}
Jul 02 14:32:41.352880 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:41.352801    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94" event={"ID":"2f52dac5-8345-47b7-9f79-b538910f1d32","Type":"ContainerStarted","Data":"45a39b073a294a0924285142f372d644ab65dfa02e6860efb225b607b3432bce"}
Jul 02 14:32:41.376437 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:41.376355    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94" podStartSLOduration=2.412193324 podStartE2EDuration="6.376335434s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.96414211s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:32:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:32:41.375259684 +0000 UTC m=+4672.603170863" watchObservedRunningTime="2026-07-02 14:32:41.376335434 +0000 UTC m=+4672.604246608"
Jul 02 14:32:42.023913 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:42.023877    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"
Jul 02 14:32:46.023943 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:46.023897    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"
Jul 02 14:32:46.025604 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:46.025564    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94" podUID="2f52dac5-8345-47b7-9f79-b538910f1d32" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 02 14:32:56.024134 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:32:56.024090    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94" podUID="2f52dac5-8345-47b7-9f79-b538910f1d32" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 02 14:33:06.024764 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:06.024711    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94" podUID="2f52dac5-8345-47b7-9f79-b538910f1d32" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 02 14:33:10.347718 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:10.347684    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx"
Jul 02 14:33:11.499410 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:11.499358    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:33:12.665544 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:12.665510    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:33:12.764637 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:12.764603    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:33:16.024194 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:16.024144    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94" podUID="2f52dac5-8345-47b7-9f79-b538910f1d32" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 02 14:33:26.023915 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:26.023865    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94" podUID="2f52dac5-8345-47b7-9f79-b538910f1d32" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 02 14:33:36.023899 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:36.023852    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94" podUID="2f52dac5-8345-47b7-9f79-b538910f1d32" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 02 14:33:38.026971 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:38.026935    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k"]
Jul 02 14:33:46.024161 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:46.024108    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94" podUID="2f52dac5-8345-47b7-9f79-b538910f1d32" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 02 14:33:51.341559 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:51.341514    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd"]
Jul 02 14:33:51.351660 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:51.351623    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:33:51.352926 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:33:51.352891    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 02 14:33:51.352926 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 14:33:51.352926 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 14:33:51.352926 ip-10-0-135-143 kubenswrapper[2583]:  > containerID="11453d1aa4a34e3c77fc7639450bd17930b6d3f59b593153682a25ffddf46998" cmd=["/bin/sleep","15"]
Jul 02 14:33:51.353275 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:51.352942    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd" message="" err=<
Jul 02 14:33:51.353275 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 14:33:51.353275 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 14:33:51.353275 ip-10-0-135-143 kubenswrapper[2583]:  >
Jul 02 14:33:51.353275 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:33:51.352956    2583 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 02 14:33:51.353275 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 14:33:51.353275 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 14:33:51.353275 ip-10-0-135-143 kubenswrapper[2583]:  > pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd" podUID="3f4e645b-25e2-4635-af7f-ff1a6b932402" containerName="main" containerID="cri-o://11453d1aa4a34e3c77fc7639450bd17930b6d3f59b593153682a25ffddf46998"
Jul 02 14:33:51.353275 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:51.353007    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd" podUID="3f4e645b-25e2-4635-af7f-ff1a6b932402" containerName="main" containerID="cri-o://11453d1aa4a34e3c77fc7639450bd17930b6d3f59b593153682a25ffddf46998" gracePeriod=60
Jul 02 14:33:51.450657 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:51.450622    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:33:51.608863 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:51.608834    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd"
Jul 02 14:33:51.659157 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:51.659112    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3f4e645b-25e2-4635-af7f-ff1a6b932402-tokenizer-uds\") pod \"3f4e645b-25e2-4635-af7f-ff1a6b932402\" (UID: \"3f4e645b-25e2-4635-af7f-ff1a6b932402\") "
Jul 02 14:33:51.659361 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:51.659171    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6x42\" (UniqueName: \"kubernetes.io/projected/3f4e645b-25e2-4635-af7f-ff1a6b932402-kube-api-access-n6x42\") pod \"3f4e645b-25e2-4635-af7f-ff1a6b932402\" (UID: \"3f4e645b-25e2-4635-af7f-ff1a6b932402\") "
Jul 02 14:33:51.659361 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:51.659235    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f4e645b-25e2-4635-af7f-ff1a6b932402-tls-certs\") pod \"3f4e645b-25e2-4635-af7f-ff1a6b932402\" (UID: \"3f4e645b-25e2-4635-af7f-ff1a6b932402\") "
Jul 02 14:33:51.659517 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:51.659441    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4e645b-25e2-4635-af7f-ff1a6b932402-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "3f4e645b-25e2-4635-af7f-ff1a6b932402" (UID: "3f4e645b-25e2-4635-af7f-ff1a6b932402"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:33:51.659581 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:51.659542    2583 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3f4e645b-25e2-4635-af7f-ff1a6b932402-tokenizer-uds\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:33:51.661423 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:51.661373    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f4e645b-25e2-4635-af7f-ff1a6b932402-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f4e645b-25e2-4635-af7f-ff1a6b932402" (UID: "3f4e645b-25e2-4635-af7f-ff1a6b932402"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:33:51.661575 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:51.661531    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f4e645b-25e2-4635-af7f-ff1a6b932402-kube-api-access-n6x42" (OuterVolumeSpecName: "kube-api-access-n6x42") pod "3f4e645b-25e2-4635-af7f-ff1a6b932402" (UID: "3f4e645b-25e2-4635-af7f-ff1a6b932402"). InnerVolumeSpecName "kube-api-access-n6x42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:33:51.735028 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:51.734988    2583 generic.go:356] "Generic (PLEG): container finished" podID="3f4e645b-25e2-4635-af7f-ff1a6b932402" containerID="11453d1aa4a34e3c77fc7639450bd17930b6d3f59b593153682a25ffddf46998" exitCode=0
Jul 02 14:33:51.735028 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:51.735035    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd" event={"ID":"3f4e645b-25e2-4635-af7f-ff1a6b932402","Type":"ContainerDied","Data":"11453d1aa4a34e3c77fc7639450bd17930b6d3f59b593153682a25ffddf46998"}
Jul 02 14:33:51.735286 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:51.735049    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd"
Jul 02 14:33:51.735286 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:51.735101    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd" event={"ID":"3f4e645b-25e2-4635-af7f-ff1a6b932402","Type":"ContainerDied","Data":"39c260191573abbcd99195109f47e234edde6122db165aa2293e33dd37113b56"}
Jul 02 14:33:51.735286 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:51.735122    2583 scope.go:117] "RemoveContainer" containerID="11453d1aa4a34e3c77fc7639450bd17930b6d3f59b593153682a25ffddf46998"
Jul 02 14:33:51.746222 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:51.746197    2583 scope.go:117] "RemoveContainer" containerID="11453d1aa4a34e3c77fc7639450bd17930b6d3f59b593153682a25ffddf46998"
Jul 02 14:33:51.746736 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:33:51.746711    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11453d1aa4a34e3c77fc7639450bd17930b6d3f59b593153682a25ffddf46998\": container with ID starting with 11453d1aa4a34e3c77fc7639450bd17930b6d3f59b593153682a25ffddf46998 not found: ID does not exist" containerID="11453d1aa4a34e3c77fc7639450bd17930b6d3f59b593153682a25ffddf46998"
Jul 02 14:33:51.746826 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:51.746744    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11453d1aa4a34e3c77fc7639450bd17930b6d3f59b593153682a25ffddf46998"} err="failed to get container status \"11453d1aa4a34e3c77fc7639450bd17930b6d3f59b593153682a25ffddf46998\": rpc error: code = NotFound desc = could not find container \"11453d1aa4a34e3c77fc7639450bd17930b6d3f59b593153682a25ffddf46998\": container with ID starting with 11453d1aa4a34e3c77fc7639450bd17930b6d3f59b593153682a25ffddf46998 not found: ID does not exist"
Jul 02 14:33:51.756864 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:51.756826    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd"]
Jul 02 14:33:51.760619 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:51.760578    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6x42\" (UniqueName: \"kubernetes.io/projected/3f4e645b-25e2-4635-af7f-ff1a6b932402-kube-api-access-n6x42\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:33:51.760619 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:51.760607    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f4e645b-25e2-4635-af7f-ff1a6b932402-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:33:51.760835 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:51.760726    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-2hqmd"]
Jul 02 14:33:53.047685 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:53.047635    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k" podUID="3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd" containerName="main" containerID="cri-o://f5838eb4a59d76112810fc1b53b5dc18e1fdd8cd903f4e3d99c834479831f2a4" gracePeriod=45
Jul 02 14:33:53.108650 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:53.108607    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k" podUID="3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": dial tcp 10.134.0.74:8000: connect: connection refused"
Jul 02 14:33:53.236933 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:53.236893    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f4e645b-25e2-4635-af7f-ff1a6b932402" path="/var/lib/kubelet/pods/3f4e645b-25e2-4635-af7f-ff1a6b932402/volumes"
Jul 02 14:33:53.302954 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:53.302885    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k"
Jul 02 14:33:53.375710 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:53.375662    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-dshm\") pod \"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd\" (UID: \"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd\") "
Jul 02 14:33:53.375710 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:53.375716    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-tmp-dir\") pod \"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd\" (UID: \"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd\") "
Jul 02 14:33:53.375987 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:53.375776    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-model-cache\") pod \"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd\" (UID: \"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd\") "
Jul 02 14:33:53.375987 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:53.375840    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-tls-certs\") pod \"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd\" (UID: \"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd\") "
Jul 02 14:33:53.375987 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:53.375877    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhqd8\" (UniqueName: \"kubernetes.io/projected/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-kube-api-access-bhqd8\") pod \"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd\" (UID: \"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd\") "
Jul 02 14:33:53.375987 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:53.375906    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-home\") pod \"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd\" (UID: \"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd\") "
Jul 02 14:33:53.376229 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:53.376013    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-model-cache" (OuterVolumeSpecName: "model-cache") pod "3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd" (UID: "3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:33:53.376229 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:53.376028    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd" (UID: "3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:33:53.376353 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:53.376263    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-tmp-dir\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:33:53.376353 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:53.376276    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-home" (OuterVolumeSpecName: "home") pod "3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd" (UID: "3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:33:53.376353 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:53.376284    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:33:53.378010 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:53.377978    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd" (UID: "3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:33:53.378010 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:53.377986    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-dshm" (OuterVolumeSpecName: "dshm") pod "3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd" (UID: "3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:33:53.378209 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:53.378039    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-kube-api-access-bhqd8" (OuterVolumeSpecName: "kube-api-access-bhqd8") pod "3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd" (UID: "3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd"). InnerVolumeSpecName "kube-api-access-bhqd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:33:53.477699 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:53.477649    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:33:53.477699 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:53.477684    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhqd8\" (UniqueName: \"kubernetes.io/projected/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-kube-api-access-bhqd8\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:33:53.477699 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:53.477694    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-home\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:33:53.477699 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:53.477702    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd-dshm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:33:53.751904 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:53.751868    2583 generic.go:356] "Generic (PLEG): container finished" podID="3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd" containerID="f5838eb4a59d76112810fc1b53b5dc18e1fdd8cd903f4e3d99c834479831f2a4" exitCode=0
Jul 02 14:33:53.752119 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:53.751907    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k" event={"ID":"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd","Type":"ContainerDied","Data":"f5838eb4a59d76112810fc1b53b5dc18e1fdd8cd903f4e3d99c834479831f2a4"}
Jul 02 14:33:53.752119 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:53.751933    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k"
Jul 02 14:33:53.752119 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:53.751999    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k" event={"ID":"3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd","Type":"ContainerDied","Data":"d8b7aa3b9b977306acd14698eb46f196a063e06e999295c25e791f6f26323d71"}
Jul 02 14:33:53.752119 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:53.752034    2583 scope.go:117] "RemoveContainer" containerID="f5838eb4a59d76112810fc1b53b5dc18e1fdd8cd903f4e3d99c834479831f2a4"
Jul 02 14:33:53.762160 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:53.762140    2583 scope.go:117] "RemoveContainer" containerID="f5838eb4a59d76112810fc1b53b5dc18e1fdd8cd903f4e3d99c834479831f2a4"
Jul 02 14:33:53.762493 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:33:53.762471    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5838eb4a59d76112810fc1b53b5dc18e1fdd8cd903f4e3d99c834479831f2a4\": container with ID starting with f5838eb4a59d76112810fc1b53b5dc18e1fdd8cd903f4e3d99c834479831f2a4 not found: ID does not exist" containerID="f5838eb4a59d76112810fc1b53b5dc18e1fdd8cd903f4e3d99c834479831f2a4"
Jul 02 14:33:53.762568 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:53.762505    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5838eb4a59d76112810fc1b53b5dc18e1fdd8cd903f4e3d99c834479831f2a4"} err="failed to get container status \"f5838eb4a59d76112810fc1b53b5dc18e1fdd8cd903f4e3d99c834479831f2a4\": rpc error: code = NotFound desc = could not find container \"f5838eb4a59d76112810fc1b53b5dc18e1fdd8cd903f4e3d99c834479831f2a4\": container with ID starting with f5838eb4a59d76112810fc1b53b5dc18e1fdd8cd903f4e3d99c834479831f2a4 not found: ID does not exist"
Jul 02 14:33:53.777496 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:53.777454    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k"]
Jul 02 14:33:53.785226 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:53.785200    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-nlw4k"]
Jul 02 14:33:55.234714 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:55.234677    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd" path="/var/lib/kubelet/pods/3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd/volumes"
Jul 02 14:33:56.024132 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:33:56.024087    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94" podUID="2f52dac5-8345-47b7-9f79-b538910f1d32" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 02 14:34:06.024367 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:06.024314    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94" podUID="2f52dac5-8345-47b7-9f79-b538910f1d32" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 02 14:34:12.069641 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:12.069551    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-66f98455b5-tvvfz"]
Jul 02 14:34:12.070169 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:12.070147    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f4e645b-25e2-4635-af7f-ff1a6b932402" containerName="main"
Jul 02 14:34:12.070255 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:12.070172    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f4e645b-25e2-4635-af7f-ff1a6b932402" containerName="main"
Jul 02 14:34:12.070255 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:12.070227    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd" containerName="main"
Jul 02 14:34:12.070255 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:12.070236    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd" containerName="main"
Jul 02 14:34:12.070428 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:12.070345    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dc36c4c-ffb6-4dd4-a6c9-9b75ce3c1ffd" containerName="main"
Jul 02 14:34:12.070428 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:12.070362    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f4e645b-25e2-4635-af7f-ff1a6b932402" containerName="main"
Jul 02 14:34:12.072464 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:12.072442    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-66f98455b5-tvvfz"
Jul 02 14:34:12.076373 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:12.076341    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 02 14:34:12.076498 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:12.076342    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-xwpbv\""
Jul 02 14:34:12.081755 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:12.081725    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-66f98455b5-tvvfz"]
Jul 02 14:34:12.156167 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:12.156132    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31249163-a16c-4153-961c-6b50def8fd9b-cert\") pod \"llmisvc-controller-manager-66f98455b5-tvvfz\" (UID: \"31249163-a16c-4153-961c-6b50def8fd9b\") " pod="kserve/llmisvc-controller-manager-66f98455b5-tvvfz"
Jul 02 14:34:12.156346 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:12.156213    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7xl2\" (UniqueName: \"kubernetes.io/projected/31249163-a16c-4153-961c-6b50def8fd9b-kube-api-access-d7xl2\") pod \"llmisvc-controller-manager-66f98455b5-tvvfz\" (UID: \"31249163-a16c-4153-961c-6b50def8fd9b\") " pod="kserve/llmisvc-controller-manager-66f98455b5-tvvfz"
Jul 02 14:34:12.256691 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:12.256654    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7xl2\" (UniqueName: \"kubernetes.io/projected/31249163-a16c-4153-961c-6b50def8fd9b-kube-api-access-d7xl2\") pod \"llmisvc-controller-manager-66f98455b5-tvvfz\" (UID: \"31249163-a16c-4153-961c-6b50def8fd9b\") " pod="kserve/llmisvc-controller-manager-66f98455b5-tvvfz"
Jul 02 14:34:12.256838 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:12.256746    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31249163-a16c-4153-961c-6b50def8fd9b-cert\") pod \"llmisvc-controller-manager-66f98455b5-tvvfz\" (UID: \"31249163-a16c-4153-961c-6b50def8fd9b\") " pod="kserve/llmisvc-controller-manager-66f98455b5-tvvfz"
Jul 02 14:34:12.259585 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:12.259560    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31249163-a16c-4153-961c-6b50def8fd9b-cert\") pod \"llmisvc-controller-manager-66f98455b5-tvvfz\" (UID: \"31249163-a16c-4153-961c-6b50def8fd9b\") " pod="kserve/llmisvc-controller-manager-66f98455b5-tvvfz"
Jul 02 14:34:12.265723 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:12.265674    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7xl2\" (UniqueName: \"kubernetes.io/projected/31249163-a16c-4153-961c-6b50def8fd9b-kube-api-access-d7xl2\") pod \"llmisvc-controller-manager-66f98455b5-tvvfz\" (UID: \"31249163-a16c-4153-961c-6b50def8fd9b\") " pod="kserve/llmisvc-controller-manager-66f98455b5-tvvfz"
Jul 02 14:34:12.390752 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:12.384815    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-66f98455b5-tvvfz"
Jul 02 14:34:12.537027 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:12.537000    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-66f98455b5-tvvfz"]
Jul 02 14:34:12.539873 ip-10-0-135-143 kubenswrapper[2583]: W0702 14:34:12.539842    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod31249163_a16c_4153_961c_6b50def8fd9b.slice/crio-247fae2057a12783cb68cd0a008a697916e491b99a786b3ee4848c8f50297317 WatchSource:0}: Error finding container 247fae2057a12783cb68cd0a008a697916e491b99a786b3ee4848c8f50297317: Status 404 returned error can't find the container with id 247fae2057a12783cb68cd0a008a697916e491b99a786b3ee4848c8f50297317
Jul 02 14:34:12.541734 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:12.541711    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 14:34:12.859183 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:12.859087    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-66f98455b5-tvvfz" event={"ID":"31249163-a16c-4153-961c-6b50def8fd9b","Type":"ContainerStarted","Data":"247fae2057a12783cb68cd0a008a697916e491b99a786b3ee4848c8f50297317"}
Jul 02 14:34:15.875633 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:15.875596    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-66f98455b5-tvvfz" event={"ID":"31249163-a16c-4153-961c-6b50def8fd9b","Type":"ContainerStarted","Data":"f8fe0835bc7acd26ae7d8f9f12bb3eb26c03d363b9c0dc24d45625b362988738"}
Jul 02 14:34:15.876044 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:15.875842    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-66f98455b5-tvvfz"
Jul 02 14:34:15.892868 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:15.892799    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-66f98455b5-tvvfz" podStartSLOduration=0.723316944 podStartE2EDuration="3.892774135s" totalImagesPullingTime="3.169457191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:34:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:34:15.892218381 +0000 UTC m=+4767.120129557" watchObservedRunningTime="2026-07-02 14:34:15.892774135 +0000 UTC m=+4767.120685314"
Jul 02 14:34:16.024200 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:16.024099    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94" podUID="2f52dac5-8345-47b7-9f79-b538910f1d32" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 02 14:34:26.024597 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:26.024550    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94" podUID="2f52dac5-8345-47b7-9f79-b538910f1d32" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 02 14:34:36.024612 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:36.024565    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94" podUID="2f52dac5-8345-47b7-9f79-b538910f1d32" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 02 14:34:38.139922 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:38.139882    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:34:46.033727 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:46.033689    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"
Jul 02 14:34:46.042275 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:46.042244    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"
Jul 02 14:34:46.883170 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:46.883134    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-66f98455b5-tvvfz"
Jul 02 14:34:47.693971 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:47.693933    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"]
Jul 02 14:34:47.720137 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:47.720093    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl"]
Jul 02 14:34:47.723682 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:47.723664    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl"
Jul 02 14:34:47.734995 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:47.734969    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl"]
Jul 02 14:34:47.896823 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:47.896779    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c32621f-bb83-4d4a-8c1d-67958ff6be61-dshm\") pod \"prestop-hook-test-kserve-c4c794bb6-td8zl\" (UID: \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl"
Jul 02 14:34:47.896823 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:47.896822    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c32621f-bb83-4d4a-8c1d-67958ff6be61-tls-certs\") pod \"prestop-hook-test-kserve-c4c794bb6-td8zl\" (UID: \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl"
Jul 02 14:34:47.897049 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:47.896897    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c32621f-bb83-4d4a-8c1d-67958ff6be61-home\") pod \"prestop-hook-test-kserve-c4c794bb6-td8zl\" (UID: \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl"
Jul 02 14:34:47.897049 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:47.896962    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c32621f-bb83-4d4a-8c1d-67958ff6be61-kserve-provision-location\") pod \"prestop-hook-test-kserve-c4c794bb6-td8zl\" (UID: \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl"
Jul 02 14:34:47.897123 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:47.897048    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q46tz\" (UniqueName: \"kubernetes.io/projected/2c32621f-bb83-4d4a-8c1d-67958ff6be61-kube-api-access-q46tz\") pod \"prestop-hook-test-kserve-c4c794bb6-td8zl\" (UID: \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl"
Jul 02 14:34:47.897123 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:47.897082    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c32621f-bb83-4d4a-8c1d-67958ff6be61-tmp-dir\") pod \"prestop-hook-test-kserve-c4c794bb6-td8zl\" (UID: \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl"
Jul 02 14:34:47.897123 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:47.897106    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c32621f-bb83-4d4a-8c1d-67958ff6be61-model-cache\") pod \"prestop-hook-test-kserve-c4c794bb6-td8zl\" (UID: \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl"
Jul 02 14:34:47.998213 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:47.998123    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c32621f-bb83-4d4a-8c1d-67958ff6be61-kserve-provision-location\") pod \"prestop-hook-test-kserve-c4c794bb6-td8zl\" (UID: \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl"
Jul 02 14:34:47.998213 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:47.998191    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q46tz\" (UniqueName: \"kubernetes.io/projected/2c32621f-bb83-4d4a-8c1d-67958ff6be61-kube-api-access-q46tz\") pod \"prestop-hook-test-kserve-c4c794bb6-td8zl\" (UID: \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl"
Jul 02 14:34:47.998213 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:47.998212    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c32621f-bb83-4d4a-8c1d-67958ff6be61-tmp-dir\") pod \"prestop-hook-test-kserve-c4c794bb6-td8zl\" (UID: \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl"
Jul 02 14:34:47.998531 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:47.998233    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c32621f-bb83-4d4a-8c1d-67958ff6be61-model-cache\") pod \"prestop-hook-test-kserve-c4c794bb6-td8zl\" (UID: \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl"
Jul 02 14:34:47.998531 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:47.998257    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c32621f-bb83-4d4a-8c1d-67958ff6be61-dshm\") pod \"prestop-hook-test-kserve-c4c794bb6-td8zl\" (UID: \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl"
Jul 02 14:34:47.998531 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:47.998274    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c32621f-bb83-4d4a-8c1d-67958ff6be61-tls-certs\") pod \"prestop-hook-test-kserve-c4c794bb6-td8zl\" (UID: \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl"
Jul 02 14:34:47.998531 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:47.998308    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c32621f-bb83-4d4a-8c1d-67958ff6be61-home\") pod \"prestop-hook-test-kserve-c4c794bb6-td8zl\" (UID: \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl"
Jul 02 14:34:47.998810 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:47.998635    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c32621f-bb83-4d4a-8c1d-67958ff6be61-kserve-provision-location\") pod \"prestop-hook-test-kserve-c4c794bb6-td8zl\" (UID: \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl"
Jul 02 14:34:47.998810 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:47.998666    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c32621f-bb83-4d4a-8c1d-67958ff6be61-tmp-dir\") pod \"prestop-hook-test-kserve-c4c794bb6-td8zl\" (UID: \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl"
Jul 02 14:34:47.998810 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:47.998728    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c32621f-bb83-4d4a-8c1d-67958ff6be61-model-cache\") pod \"prestop-hook-test-kserve-c4c794bb6-td8zl\" (UID: \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl"
Jul 02 14:34:47.998810 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:47.998758    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c32621f-bb83-4d4a-8c1d-67958ff6be61-home\") pod \"prestop-hook-test-kserve-c4c794bb6-td8zl\" (UID: \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl"
Jul 02 14:34:48.000703 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:48.000680    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c32621f-bb83-4d4a-8c1d-67958ff6be61-dshm\") pod \"prestop-hook-test-kserve-c4c794bb6-td8zl\" (UID: \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl"
Jul 02 14:34:48.000858 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:48.000841    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c32621f-bb83-4d4a-8c1d-67958ff6be61-tls-certs\") pod \"prestop-hook-test-kserve-c4c794bb6-td8zl\" (UID: \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl"
Jul 02 14:34:48.007219 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:48.007194    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q46tz\" (UniqueName: \"kubernetes.io/projected/2c32621f-bb83-4d4a-8c1d-67958ff6be61-kube-api-access-q46tz\") pod \"prestop-hook-test-kserve-c4c794bb6-td8zl\" (UID: \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl"
Jul 02 14:34:48.034558 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:48.034512    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl"
Jul 02 14:34:48.174287 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:48.174260    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl"]
Jul 02 14:34:48.176583 ip-10-0-135-143 kubenswrapper[2583]: W0702 14:34:48.176556    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c32621f_bb83_4d4a_8c1d_67958ff6be61.slice/crio-f25118b92bd3aea319947802e0588067f0ac1b1a983f039262019482eb201a82 WatchSource:0}: Error finding container f25118b92bd3aea319947802e0588067f0ac1b1a983f039262019482eb201a82: Status 404 returned error can't find the container with id f25118b92bd3aea319947802e0588067f0ac1b1a983f039262019482eb201a82
Jul 02 14:34:49.063838 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:49.063776    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl" event={"ID":"2c32621f-bb83-4d4a-8c1d-67958ff6be61","Type":"ContainerStarted","Data":"e381848c32a02c3b735cf502148dc4889c5dc11e1692f2a5c329e9f3273e81f3"}
Jul 02 14:34:49.064241 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:49.063860    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl" event={"ID":"2c32621f-bb83-4d4a-8c1d-67958ff6be61","Type":"ContainerStarted","Data":"f25118b92bd3aea319947802e0588067f0ac1b1a983f039262019482eb201a82"}
Jul 02 14:34:49.711916 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:49.711801    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 14:34:49.734603 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:49.734567    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jlkds_f0256660-1798-4dba-a311-3705cb73c9ca/ovn-acl-logging/0.log"
Jul 02 14:34:52.083191 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:52.083152    2583 generic.go:356] "Generic (PLEG): container finished" podID="2c32621f-bb83-4d4a-8c1d-67958ff6be61" containerID="e381848c32a02c3b735cf502148dc4889c5dc11e1692f2a5c329e9f3273e81f3" exitCode=0
Jul 02 14:34:52.083632 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:52.083218    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl" event={"ID":"2c32621f-bb83-4d4a-8c1d-67958ff6be61","Type":"ContainerDied","Data":"e381848c32a02c3b735cf502148dc4889c5dc11e1692f2a5c329e9f3273e81f3"}
Jul 02 14:34:53.092615 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:53.092579    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl" event={"ID":"2c32621f-bb83-4d4a-8c1d-67958ff6be61","Type":"ContainerStarted","Data":"09cbe552517f47c09849c44771f6e51a9a8ca76ba21a2bbdc3f95c2abcd1dfd2"}
Jul 02 14:34:53.115623 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:53.115560    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl" podStartSLOduration=2.463746555 podStartE2EDuration="6.115538552s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.651791997s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:34:53.115279366 +0000 UTC m=+4804.343190540" watchObservedRunningTime="2026-07-02 14:34:53.115538552 +0000 UTC m=+4804.343449727"
Jul 02 14:34:54.034880 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:54.034838    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl"
Jul 02 14:34:58.034885 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:58.034850    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl"
Jul 02 14:34:58.036667 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:34:58.036637    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl" podUID="2c32621f-bb83-4d4a-8c1d-67958ff6be61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Jul 02 14:35:02.718022 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:02.717978    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94" podUID="2f52dac5-8345-47b7-9f79-b538910f1d32" containerName="main" containerID="cri-o://45a39b073a294a0924285142f372d644ab65dfa02e6860efb225b607b3432bce" gracePeriod=45
Jul 02 14:35:03.034844 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:03.034743    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94" podUID="2f52dac5-8345-47b7-9f79-b538910f1d32" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 02 14:35:03.513737 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:03.513699    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"
Jul 02 14:35:03.564716 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:03.564680    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f52dac5-8345-47b7-9f79-b538910f1d32-dshm\") pod \"2f52dac5-8345-47b7-9f79-b538910f1d32\" (UID: \"2f52dac5-8345-47b7-9f79-b538910f1d32\") "
Jul 02 14:35:03.564918 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:03.564738    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f52dac5-8345-47b7-9f79-b538910f1d32-tmp-dir\") pod \"2f52dac5-8345-47b7-9f79-b538910f1d32\" (UID: \"2f52dac5-8345-47b7-9f79-b538910f1d32\") "
Jul 02 14:35:03.564918 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:03.564783    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f52dac5-8345-47b7-9f79-b538910f1d32-tls-certs\") pod \"2f52dac5-8345-47b7-9f79-b538910f1d32\" (UID: \"2f52dac5-8345-47b7-9f79-b538910f1d32\") "
Jul 02 14:35:03.564918 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:03.564820    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f52dac5-8345-47b7-9f79-b538910f1d32-home\") pod \"2f52dac5-8345-47b7-9f79-b538910f1d32\" (UID: \"2f52dac5-8345-47b7-9f79-b538910f1d32\") "
Jul 02 14:35:03.564918 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:03.564845    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d86ds\" (UniqueName: \"kubernetes.io/projected/2f52dac5-8345-47b7-9f79-b538910f1d32-kube-api-access-d86ds\") pod \"2f52dac5-8345-47b7-9f79-b538910f1d32\" (UID: \"2f52dac5-8345-47b7-9f79-b538910f1d32\") "
Jul 02 14:35:03.564918 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:03.564873    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f52dac5-8345-47b7-9f79-b538910f1d32-kserve-provision-location\") pod \"2f52dac5-8345-47b7-9f79-b538910f1d32\" (UID: \"2f52dac5-8345-47b7-9f79-b538910f1d32\") "
Jul 02 14:35:03.564918 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:03.564910    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f52dac5-8345-47b7-9f79-b538910f1d32-model-cache\") pod \"2f52dac5-8345-47b7-9f79-b538910f1d32\" (UID: \"2f52dac5-8345-47b7-9f79-b538910f1d32\") "
Jul 02 14:35:03.565466 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:03.565437    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f52dac5-8345-47b7-9f79-b538910f1d32-model-cache" (OuterVolumeSpecName: "model-cache") pod "2f52dac5-8345-47b7-9f79-b538910f1d32" (UID: "2f52dac5-8345-47b7-9f79-b538910f1d32"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:35:03.565673 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:03.565482    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f52dac5-8345-47b7-9f79-b538910f1d32-home" (OuterVolumeSpecName: "home") pod "2f52dac5-8345-47b7-9f79-b538910f1d32" (UID: "2f52dac5-8345-47b7-9f79-b538910f1d32"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:35:03.567781 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:03.567364    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f52dac5-8345-47b7-9f79-b538910f1d32-dshm" (OuterVolumeSpecName: "dshm") pod "2f52dac5-8345-47b7-9f79-b538910f1d32" (UID: "2f52dac5-8345-47b7-9f79-b538910f1d32"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:35:03.567781 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:03.567531    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f52dac5-8345-47b7-9f79-b538910f1d32-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2f52dac5-8345-47b7-9f79-b538910f1d32" (UID: "2f52dac5-8345-47b7-9f79-b538910f1d32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:35:03.568156 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:03.568131    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f52dac5-8345-47b7-9f79-b538910f1d32-kube-api-access-d86ds" (OuterVolumeSpecName: "kube-api-access-d86ds") pod "2f52dac5-8345-47b7-9f79-b538910f1d32" (UID: "2f52dac5-8345-47b7-9f79-b538910f1d32"). InnerVolumeSpecName "kube-api-access-d86ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:35:03.578652 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:03.578606    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f52dac5-8345-47b7-9f79-b538910f1d32-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2f52dac5-8345-47b7-9f79-b538910f1d32" (UID: "2f52dac5-8345-47b7-9f79-b538910f1d32"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:35:03.647577 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:03.647532    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f52dac5-8345-47b7-9f79-b538910f1d32-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2f52dac5-8345-47b7-9f79-b538910f1d32" (UID: "2f52dac5-8345-47b7-9f79-b538910f1d32"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:35:03.666446 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:03.666409    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f52dac5-8345-47b7-9f79-b538910f1d32-tmp-dir\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:35:03.666446 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:03.666444    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f52dac5-8345-47b7-9f79-b538910f1d32-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:35:03.666446 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:03.666455    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f52dac5-8345-47b7-9f79-b538910f1d32-home\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:35:03.666699 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:03.666470    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d86ds\" (UniqueName: \"kubernetes.io/projected/2f52dac5-8345-47b7-9f79-b538910f1d32-kube-api-access-d86ds\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:35:03.666699 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:03.666483    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f52dac5-8345-47b7-9f79-b538910f1d32-kserve-provision-location\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:35:03.666699 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:03.666494    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f52dac5-8345-47b7-9f79-b538910f1d32-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:35:03.666699 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:03.666504    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f52dac5-8345-47b7-9f79-b538910f1d32-dshm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:35:04.153675 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:04.153637    2583 generic.go:356] "Generic (PLEG): container finished" podID="2f52dac5-8345-47b7-9f79-b538910f1d32" containerID="45a39b073a294a0924285142f372d644ab65dfa02e6860efb225b607b3432bce" exitCode=0
Jul 02 14:35:04.154062 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:04.153686    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"
Jul 02 14:35:04.154062 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:04.153732    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94" event={"ID":"2f52dac5-8345-47b7-9f79-b538910f1d32","Type":"ContainerDied","Data":"45a39b073a294a0924285142f372d644ab65dfa02e6860efb225b607b3432bce"}
Jul 02 14:35:04.154062 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:04.153804    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94" event={"ID":"2f52dac5-8345-47b7-9f79-b538910f1d32","Type":"ContainerDied","Data":"c9eb739919d14326627a4b8e1a848ff9de6e1be6b592ce46051674807ea2e034"}
Jul 02 14:35:04.154062 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:04.153837    2583 scope.go:117] "RemoveContainer" containerID="45a39b073a294a0924285142f372d644ab65dfa02e6860efb225b607b3432bce"
Jul 02 14:35:04.165620 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:04.165594    2583 scope.go:117] "RemoveContainer" containerID="c7cc18336120a246b1b28f55e41f95b393a2c632d974fb3caa0be5fcb9f805fa"
Jul 02 14:35:04.176735 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:04.176698    2583 scope.go:117] "RemoveContainer" containerID="45a39b073a294a0924285142f372d644ab65dfa02e6860efb225b607b3432bce"
Jul 02 14:35:04.177098 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:35:04.177076    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45a39b073a294a0924285142f372d644ab65dfa02e6860efb225b607b3432bce\": container with ID starting with 45a39b073a294a0924285142f372d644ab65dfa02e6860efb225b607b3432bce not found: ID does not exist" containerID="45a39b073a294a0924285142f372d644ab65dfa02e6860efb225b607b3432bce"
Jul 02 14:35:04.177213 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:04.177107    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a39b073a294a0924285142f372d644ab65dfa02e6860efb225b607b3432bce"} err="failed to get container status \"45a39b073a294a0924285142f372d644ab65dfa02e6860efb225b607b3432bce\": rpc error: code = NotFound desc = could not find container \"45a39b073a294a0924285142f372d644ab65dfa02e6860efb225b607b3432bce\": container with ID starting with 45a39b073a294a0924285142f372d644ab65dfa02e6860efb225b607b3432bce not found: ID does not exist"
Jul 02 14:35:04.177213 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:04.177125    2583 scope.go:117] "RemoveContainer" containerID="c7cc18336120a246b1b28f55e41f95b393a2c632d974fb3caa0be5fcb9f805fa"
Jul 02 14:35:04.177448 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:35:04.177406    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7cc18336120a246b1b28f55e41f95b393a2c632d974fb3caa0be5fcb9f805fa\": container with ID starting with c7cc18336120a246b1b28f55e41f95b393a2c632d974fb3caa0be5fcb9f805fa not found: ID does not exist" containerID="c7cc18336120a246b1b28f55e41f95b393a2c632d974fb3caa0be5fcb9f805fa"
Jul 02 14:35:04.177448 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:04.177435    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7cc18336120a246b1b28f55e41f95b393a2c632d974fb3caa0be5fcb9f805fa"} err="failed to get container status \"c7cc18336120a246b1b28f55e41f95b393a2c632d974fb3caa0be5fcb9f805fa\": rpc error: code = NotFound desc = could not find container \"c7cc18336120a246b1b28f55e41f95b393a2c632d974fb3caa0be5fcb9f805fa\": container with ID starting with c7cc18336120a246b1b28f55e41f95b393a2c632d974fb3caa0be5fcb9f805fa not found: ID does not exist"
Jul 02 14:35:04.178069 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:04.178046    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"]
Jul 02 14:35:04.181547 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:04.181519    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-dts94"]
Jul 02 14:35:05.234264 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:05.234217    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f52dac5-8345-47b7-9f79-b538910f1d32" path="/var/lib/kubelet/pods/2f52dac5-8345-47b7-9f79-b538910f1d32/volumes"
Jul 02 14:35:08.035405 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:08.035341    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl" podUID="2c32621f-bb83-4d4a-8c1d-67958ff6be61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Jul 02 14:35:18.035795 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:18.035741    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl" podUID="2c32621f-bb83-4d4a-8c1d-67958ff6be61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Jul 02 14:35:28.035689 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:28.035636    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl" podUID="2c32621f-bb83-4d4a-8c1d-67958ff6be61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Jul 02 14:35:38.035830 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:38.035772    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl" podUID="2c32621f-bb83-4d4a-8c1d-67958ff6be61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Jul 02 14:35:48.035873 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:48.035779    2583 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl" podUID="2c32621f-bb83-4d4a-8c1d-67958ff6be61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Jul 02 14:35:48.388345 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:48.388308    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx"]
Jul 02 14:35:48.390734 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:48.390702    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:35:48.397219 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:48.397186    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl"]
Jul 02 14:35:48.401716 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:35:48.401680    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 02 14:35:48.401716 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 14:35:48.401716 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 14:35:48.401716 ip-10-0-135-143 kubenswrapper[2583]:  > containerID="fbe904c249c46a199cca2a62c102cf302cce39c9e3e72e9cae1dcd7fc30b1ee9" cmd=["/bin/sleep","15"]
Jul 02 14:35:48.402038 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:48.401733    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx" message="" err=<
Jul 02 14:35:48.402038 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 14:35:48.402038 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 14:35:48.402038 ip-10-0-135-143 kubenswrapper[2583]:  >
Jul 02 14:35:48.402038 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:35:48.401749    2583 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 02 14:35:48.402038 ip-10-0-135-143 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 02 14:35:48.402038 ip-10-0-135-143 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Jul 02 14:35:48.402038 ip-10-0-135-143 kubenswrapper[2583]:  > pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx" podUID="65cf02af-a227-41ed-9445-9de19fad5cc9" containerName="main" containerID="cri-o://fbe904c249c46a199cca2a62c102cf302cce39c9e3e72e9cae1dcd7fc30b1ee9"
Jul 02 14:35:48.402038 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:48.401805    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx" podUID="65cf02af-a227-41ed-9445-9de19fad5cc9" containerName="main" containerID="cri-o://fbe904c249c46a199cca2a62c102cf302cce39c9e3e72e9cae1dcd7fc30b1ee9" gracePeriod=60
Jul 02 14:35:48.454319 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:48.454279    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:35:48.663404 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:48.663362    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx"
Jul 02 14:35:48.820321 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:48.820274    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65cf02af-a227-41ed-9445-9de19fad5cc9-kserve-provision-location\") pod \"65cf02af-a227-41ed-9445-9de19fad5cc9\" (UID: \"65cf02af-a227-41ed-9445-9de19fad5cc9\") "
Jul 02 14:35:48.820321 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:48.820317    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/65cf02af-a227-41ed-9445-9de19fad5cc9-tokenizer-uds\") pod \"65cf02af-a227-41ed-9445-9de19fad5cc9\" (UID: \"65cf02af-a227-41ed-9445-9de19fad5cc9\") "
Jul 02 14:35:48.820618 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:48.820342    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65cf02af-a227-41ed-9445-9de19fad5cc9-tls-certs\") pod \"65cf02af-a227-41ed-9445-9de19fad5cc9\" (UID: \"65cf02af-a227-41ed-9445-9de19fad5cc9\") "
Jul 02 14:35:48.820618 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:48.820381    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2j5rh\" (UniqueName: \"kubernetes.io/projected/65cf02af-a227-41ed-9445-9de19fad5cc9-kube-api-access-2j5rh\") pod \"65cf02af-a227-41ed-9445-9de19fad5cc9\" (UID: \"65cf02af-a227-41ed-9445-9de19fad5cc9\") "
Jul 02 14:35:48.820618 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:48.820600    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65cf02af-a227-41ed-9445-9de19fad5cc9-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "65cf02af-a227-41ed-9445-9de19fad5cc9" (UID: "65cf02af-a227-41ed-9445-9de19fad5cc9"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:35:48.820798 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:48.820714    2583 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/65cf02af-a227-41ed-9445-9de19fad5cc9-tokenizer-uds\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:35:48.821092 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:48.821062    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65cf02af-a227-41ed-9445-9de19fad5cc9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "65cf02af-a227-41ed-9445-9de19fad5cc9" (UID: "65cf02af-a227-41ed-9445-9de19fad5cc9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:35:48.822614 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:48.822588    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65cf02af-a227-41ed-9445-9de19fad5cc9-kube-api-access-2j5rh" (OuterVolumeSpecName: "kube-api-access-2j5rh") pod "65cf02af-a227-41ed-9445-9de19fad5cc9" (UID: "65cf02af-a227-41ed-9445-9de19fad5cc9"). InnerVolumeSpecName "kube-api-access-2j5rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:35:48.822744 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:48.822700    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65cf02af-a227-41ed-9445-9de19fad5cc9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "65cf02af-a227-41ed-9445-9de19fad5cc9" (UID: "65cf02af-a227-41ed-9445-9de19fad5cc9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:35:48.922257 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:48.922151    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65cf02af-a227-41ed-9445-9de19fad5cc9-kserve-provision-location\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:35:48.922257 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:48.922184    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65cf02af-a227-41ed-9445-9de19fad5cc9-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:35:48.922257 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:48.922193    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j5rh\" (UniqueName: \"kubernetes.io/projected/65cf02af-a227-41ed-9445-9de19fad5cc9-kube-api-access-2j5rh\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:35:49.415173 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:49.415131    2583 generic.go:356] "Generic (PLEG): container finished" podID="65cf02af-a227-41ed-9445-9de19fad5cc9" containerID="fbe904c249c46a199cca2a62c102cf302cce39c9e3e72e9cae1dcd7fc30b1ee9" exitCode=0
Jul 02 14:35:49.415592 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:49.415206    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx"
Jul 02 14:35:49.415592 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:49.415223    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx" event={"ID":"65cf02af-a227-41ed-9445-9de19fad5cc9","Type":"ContainerDied","Data":"fbe904c249c46a199cca2a62c102cf302cce39c9e3e72e9cae1dcd7fc30b1ee9"}
Jul 02 14:35:49.415592 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:49.415306    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx" event={"ID":"65cf02af-a227-41ed-9445-9de19fad5cc9","Type":"ContainerDied","Data":"3dc7738dfbd820972bea04f498d94a92e2c9ef1356e745b40962dcafd144686d"}
Jul 02 14:35:49.415592 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:49.415327    2583 scope.go:117] "RemoveContainer" containerID="fbe904c249c46a199cca2a62c102cf302cce39c9e3e72e9cae1dcd7fc30b1ee9"
Jul 02 14:35:49.424693 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:49.424671    2583 scope.go:117] "RemoveContainer" containerID="92c6cea72dc420c9c31e62b66e151be53bd2cf81eed5a2ca526a1bdbc18f04b6"
Jul 02 14:35:49.434712 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:49.434682    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx"]
Jul 02 14:35:49.434947 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:49.434925    2583 scope.go:117] "RemoveContainer" containerID="fbe904c249c46a199cca2a62c102cf302cce39c9e3e72e9cae1dcd7fc30b1ee9"
Jul 02 14:35:49.435251 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:35:49.435233    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbe904c249c46a199cca2a62c102cf302cce39c9e3e72e9cae1dcd7fc30b1ee9\": container with ID starting with fbe904c249c46a199cca2a62c102cf302cce39c9e3e72e9cae1dcd7fc30b1ee9 not found: ID does not exist" containerID="fbe904c249c46a199cca2a62c102cf302cce39c9e3e72e9cae1dcd7fc30b1ee9"
Jul 02 14:35:49.435320 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:49.435260    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbe904c249c46a199cca2a62c102cf302cce39c9e3e72e9cae1dcd7fc30b1ee9"} err="failed to get container status \"fbe904c249c46a199cca2a62c102cf302cce39c9e3e72e9cae1dcd7fc30b1ee9\": rpc error: code = NotFound desc = could not find container \"fbe904c249c46a199cca2a62c102cf302cce39c9e3e72e9cae1dcd7fc30b1ee9\": container with ID starting with fbe904c249c46a199cca2a62c102cf302cce39c9e3e72e9cae1dcd7fc30b1ee9 not found: ID does not exist"
Jul 02 14:35:49.435320 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:49.435278    2583 scope.go:117] "RemoveContainer" containerID="92c6cea72dc420c9c31e62b66e151be53bd2cf81eed5a2ca526a1bdbc18f04b6"
Jul 02 14:35:49.435881 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:35:49.435777    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92c6cea72dc420c9c31e62b66e151be53bd2cf81eed5a2ca526a1bdbc18f04b6\": container with ID starting with 92c6cea72dc420c9c31e62b66e151be53bd2cf81eed5a2ca526a1bdbc18f04b6 not found: ID does not exist" containerID="92c6cea72dc420c9c31e62b66e151be53bd2cf81eed5a2ca526a1bdbc18f04b6"
Jul 02 14:35:49.435881 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:49.435852    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92c6cea72dc420c9c31e62b66e151be53bd2cf81eed5a2ca526a1bdbc18f04b6"} err="failed to get container status \"92c6cea72dc420c9c31e62b66e151be53bd2cf81eed5a2ca526a1bdbc18f04b6\": rpc error: code = NotFound desc = could not find container \"92c6cea72dc420c9c31e62b66e151be53bd2cf81eed5a2ca526a1bdbc18f04b6\": container with ID starting with 92c6cea72dc420c9c31e62b66e151be53bd2cf81eed5a2ca526a1bdbc18f04b6 not found: ID does not exist"
Jul 02 14:35:49.438115 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:49.438090    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-547bcf85cf-bj9cx"]
Jul 02 14:35:51.234110 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:35:51.234077    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65cf02af-a227-41ed-9445-9de19fad5cc9" path="/var/lib/kubelet/pods/65cf02af-a227-41ed-9445-9de19fad5cc9/volumes"
Jul 02 14:36:03.430802 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:03.430756    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl" podUID="2c32621f-bb83-4d4a-8c1d-67958ff6be61" containerName="main" containerID="cri-o://09cbe552517f47c09849c44771f6e51a9a8ca76ba21a2bbdc3f95c2abcd1dfd2" gracePeriod=45
Jul 02 14:36:48.711309 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:48.711227    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_prestop-hook-test-kserve-c4c794bb6-td8zl_2c32621f-bb83-4d4a-8c1d-67958ff6be61/main/0.log"
Jul 02 14:36:48.711786 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:48.711629    2583 generic.go:356] "Generic (PLEG): container finished" podID="2c32621f-bb83-4d4a-8c1d-67958ff6be61" containerID="09cbe552517f47c09849c44771f6e51a9a8ca76ba21a2bbdc3f95c2abcd1dfd2" exitCode=137
Jul 02 14:36:48.711786 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:48.711707    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl" event={"ID":"2c32621f-bb83-4d4a-8c1d-67958ff6be61","Type":"ContainerDied","Data":"09cbe552517f47c09849c44771f6e51a9a8ca76ba21a2bbdc3f95c2abcd1dfd2"}
Jul 02 14:36:49.084561 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:49.084535    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_prestop-hook-test-kserve-c4c794bb6-td8zl_2c32621f-bb83-4d4a-8c1d-67958ff6be61/main/0.log"
Jul 02 14:36:49.087117 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:49.087098    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl"
Jul 02 14:36:49.208285 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:49.208247    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c32621f-bb83-4d4a-8c1d-67958ff6be61-tmp-dir\") pod \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\" (UID: \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\") "
Jul 02 14:36:49.208509 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:49.208309    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c32621f-bb83-4d4a-8c1d-67958ff6be61-dshm\") pod \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\" (UID: \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\") "
Jul 02 14:36:49.208509 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:49.208340    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c32621f-bb83-4d4a-8c1d-67958ff6be61-model-cache\") pod \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\" (UID: \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\") "
Jul 02 14:36:49.208509 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:49.208444    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c32621f-bb83-4d4a-8c1d-67958ff6be61-home\") pod \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\" (UID: \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\") "
Jul 02 14:36:49.208509 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:49.208473    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c32621f-bb83-4d4a-8c1d-67958ff6be61-tls-certs\") pod \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\" (UID: \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\") "
Jul 02 14:36:49.208509 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:49.208498    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c32621f-bb83-4d4a-8c1d-67958ff6be61-kserve-provision-location\") pod \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\" (UID: \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\") "
Jul 02 14:36:49.208788 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:49.208563    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q46tz\" (UniqueName: \"kubernetes.io/projected/2c32621f-bb83-4d4a-8c1d-67958ff6be61-kube-api-access-q46tz\") pod \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\" (UID: \"2c32621f-bb83-4d4a-8c1d-67958ff6be61\") "
Jul 02 14:36:49.208788 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:49.208735    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c32621f-bb83-4d4a-8c1d-67958ff6be61-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2c32621f-bb83-4d4a-8c1d-67958ff6be61" (UID: "2c32621f-bb83-4d4a-8c1d-67958ff6be61"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:36:49.208788 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:49.208752    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c32621f-bb83-4d4a-8c1d-67958ff6be61-model-cache" (OuterVolumeSpecName: "model-cache") pod "2c32621f-bb83-4d4a-8c1d-67958ff6be61" (UID: "2c32621f-bb83-4d4a-8c1d-67958ff6be61"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:36:49.208950 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:49.208926    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c32621f-bb83-4d4a-8c1d-67958ff6be61-home" (OuterVolumeSpecName: "home") pod "2c32621f-bb83-4d4a-8c1d-67958ff6be61" (UID: "2c32621f-bb83-4d4a-8c1d-67958ff6be61"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:36:49.208994 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:49.208945    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c32621f-bb83-4d4a-8c1d-67958ff6be61-model-cache\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:36:49.208994 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:49.208964    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c32621f-bb83-4d4a-8c1d-67958ff6be61-tmp-dir\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:36:49.210869 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:49.210841    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c32621f-bb83-4d4a-8c1d-67958ff6be61-dshm" (OuterVolumeSpecName: "dshm") pod "2c32621f-bb83-4d4a-8c1d-67958ff6be61" (UID: "2c32621f-bb83-4d4a-8c1d-67958ff6be61"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:36:49.211104 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:49.211072    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c32621f-bb83-4d4a-8c1d-67958ff6be61-kube-api-access-q46tz" (OuterVolumeSpecName: "kube-api-access-q46tz") pod "2c32621f-bb83-4d4a-8c1d-67958ff6be61" (UID: "2c32621f-bb83-4d4a-8c1d-67958ff6be61"). InnerVolumeSpecName "kube-api-access-q46tz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:36:49.211227 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:49.211139    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c32621f-bb83-4d4a-8c1d-67958ff6be61-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2c32621f-bb83-4d4a-8c1d-67958ff6be61" (UID: "2c32621f-bb83-4d4a-8c1d-67958ff6be61"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:36:49.267858 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:49.267736    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c32621f-bb83-4d4a-8c1d-67958ff6be61-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2c32621f-bb83-4d4a-8c1d-67958ff6be61" (UID: "2c32621f-bb83-4d4a-8c1d-67958ff6be61"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:36:49.309763 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:49.309734    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q46tz\" (UniqueName: \"kubernetes.io/projected/2c32621f-bb83-4d4a-8c1d-67958ff6be61-kube-api-access-q46tz\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:36:49.309763 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:49.309763    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c32621f-bb83-4d4a-8c1d-67958ff6be61-dshm\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:36:49.309763 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:49.309772    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c32621f-bb83-4d4a-8c1d-67958ff6be61-home\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:36:49.309998 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:49.309780    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c32621f-bb83-4d4a-8c1d-67958ff6be61-tls-certs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:36:49.309998 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:49.309807    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c32621f-bb83-4d4a-8c1d-67958ff6be61-kserve-provision-location\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:36:49.718105 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:49.718077    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_prestop-hook-test-kserve-c4c794bb6-td8zl_2c32621f-bb83-4d4a-8c1d-67958ff6be61/main/0.log"
Jul 02 14:36:49.718532 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:49.718451    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl" event={"ID":"2c32621f-bb83-4d4a-8c1d-67958ff6be61","Type":"ContainerDied","Data":"f25118b92bd3aea319947802e0588067f0ac1b1a983f039262019482eb201a82"}
Jul 02 14:36:49.718532 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:49.718486    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl"
Jul 02 14:36:49.718532 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:49.718505    2583 scope.go:117] "RemoveContainer" containerID="09cbe552517f47c09849c44771f6e51a9a8ca76ba21a2bbdc3f95c2abcd1dfd2"
Jul 02 14:36:49.727509 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:49.727486    2583 scope.go:117] "RemoveContainer" containerID="e381848c32a02c3b735cf502148dc4889c5dc11e1692f2a5c329e9f3273e81f3"
Jul 02 14:36:49.742731 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:49.742702    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl"]
Jul 02 14:36:49.746878 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:49.746853    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-c4c794bb6-td8zl"]
Jul 02 14:36:51.234212 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:51.234178    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c32621f-bb83-4d4a-8c1d-67958ff6be61" path="/var/lib/kubelet/pods/2c32621f-bb83-4d4a-8c1d-67958ff6be61/volumes"
Jul 02 14:36:54.332531 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:54.332493    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bkmqr/must-gather-9thgl"]
Jul 02 14:36:54.333178 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:54.332916    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f52dac5-8345-47b7-9f79-b538910f1d32" containerName="main"
Jul 02 14:36:54.333178 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:54.332928    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f52dac5-8345-47b7-9f79-b538910f1d32" containerName="main"
Jul 02 14:36:54.333178 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:54.332938    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c32621f-bb83-4d4a-8c1d-67958ff6be61" containerName="main"
Jul 02 14:36:54.333178 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:54.332943    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c32621f-bb83-4d4a-8c1d-67958ff6be61" containerName="main"
Jul 02 14:36:54.333178 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:54.332958    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65cf02af-a227-41ed-9445-9de19fad5cc9" containerName="main"
Jul 02 14:36:54.333178 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:54.332964    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="65cf02af-a227-41ed-9445-9de19fad5cc9" containerName="main"
Jul 02 14:36:54.333178 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:54.332974    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c32621f-bb83-4d4a-8c1d-67958ff6be61" containerName="storage-initializer"
Jul 02 14:36:54.333178 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:54.332979    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c32621f-bb83-4d4a-8c1d-67958ff6be61" containerName="storage-initializer"
Jul 02 14:36:54.333178 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:54.332987    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f52dac5-8345-47b7-9f79-b538910f1d32" containerName="storage-initializer"
Jul 02 14:36:54.333178 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:54.332991    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f52dac5-8345-47b7-9f79-b538910f1d32" containerName="storage-initializer"
Jul 02 14:36:54.333178 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:54.333000    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65cf02af-a227-41ed-9445-9de19fad5cc9" containerName="storage-initializer"
Jul 02 14:36:54.333178 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:54.333005    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="65cf02af-a227-41ed-9445-9de19fad5cc9" containerName="storage-initializer"
Jul 02 14:36:54.333178 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:54.333057    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="65cf02af-a227-41ed-9445-9de19fad5cc9" containerName="main"
Jul 02 14:36:54.333178 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:54.333064    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c32621f-bb83-4d4a-8c1d-67958ff6be61" containerName="main"
Jul 02 14:36:54.333178 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:54.333073    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f52dac5-8345-47b7-9f79-b538910f1d32" containerName="main"
Jul 02 14:36:54.337800 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:54.337782    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bkmqr/must-gather-9thgl"
Jul 02 14:36:54.340704 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:54.340680    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bkmqr\"/\"kube-root-ca.crt\""
Jul 02 14:36:54.340861 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:54.340737    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bkmqr\"/\"openshift-service-ca.crt\""
Jul 02 14:36:54.346729 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:54.346700    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bkmqr/must-gather-9thgl"]
Jul 02 14:36:54.456064 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:54.456021    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/65dae769-a7e6-454a-bb17-beed3dfc2caf-must-gather-output\") pod \"must-gather-9thgl\" (UID: \"65dae769-a7e6-454a-bb17-beed3dfc2caf\") " pod="openshift-must-gather-bkmqr/must-gather-9thgl"
Jul 02 14:36:54.456247 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:54.456149    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k5xs\" (UniqueName: \"kubernetes.io/projected/65dae769-a7e6-454a-bb17-beed3dfc2caf-kube-api-access-9k5xs\") pod \"must-gather-9thgl\" (UID: \"65dae769-a7e6-454a-bb17-beed3dfc2caf\") " pod="openshift-must-gather-bkmqr/must-gather-9thgl"
Jul 02 14:36:54.495177 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:54.495142    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:36:54.557190 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:54.557152    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k5xs\" (UniqueName: \"kubernetes.io/projected/65dae769-a7e6-454a-bb17-beed3dfc2caf-kube-api-access-9k5xs\") pod \"must-gather-9thgl\" (UID: \"65dae769-a7e6-454a-bb17-beed3dfc2caf\") " pod="openshift-must-gather-bkmqr/must-gather-9thgl"
Jul 02 14:36:54.557371 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:54.557214    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/65dae769-a7e6-454a-bb17-beed3dfc2caf-must-gather-output\") pod \"must-gather-9thgl\" (UID: \"65dae769-a7e6-454a-bb17-beed3dfc2caf\") " pod="openshift-must-gather-bkmqr/must-gather-9thgl"
Jul 02 14:36:54.557629 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:54.557609    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/65dae769-a7e6-454a-bb17-beed3dfc2caf-must-gather-output\") pod \"must-gather-9thgl\" (UID: \"65dae769-a7e6-454a-bb17-beed3dfc2caf\") " pod="openshift-must-gather-bkmqr/must-gather-9thgl"
Jul 02 14:36:54.569118 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:54.569083    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k5xs\" (UniqueName: \"kubernetes.io/projected/65dae769-a7e6-454a-bb17-beed3dfc2caf-kube-api-access-9k5xs\") pod \"must-gather-9thgl\" (UID: \"65dae769-a7e6-454a-bb17-beed3dfc2caf\") " pod="openshift-must-gather-bkmqr/must-gather-9thgl"
Jul 02 14:36:54.648464 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:54.648415    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bkmqr/must-gather-9thgl"
Jul 02 14:36:54.791788 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:54.791581    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bkmqr/must-gather-9thgl"]
Jul 02 14:36:54.793959 ip-10-0-135-143 kubenswrapper[2583]: W0702 14:36:54.793931    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65dae769_a7e6_454a_bb17_beed3dfc2caf.slice/crio-fcfbaff40482abe2c0a8bb62c3a6a58c0a3a5247fe56ccc8c24fc908e7e2ebcb WatchSource:0}: Error finding container fcfbaff40482abe2c0a8bb62c3a6a58c0a3a5247fe56ccc8c24fc908e7e2ebcb: Status 404 returned error can't find the container with id fcfbaff40482abe2c0a8bb62c3a6a58c0a3a5247fe56ccc8c24fc908e7e2ebcb
Jul 02 14:36:55.749100 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:55.749062    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bkmqr/must-gather-9thgl" event={"ID":"65dae769-a7e6-454a-bb17-beed3dfc2caf","Type":"ContainerStarted","Data":"fcfbaff40482abe2c0a8bb62c3a6a58c0a3a5247fe56ccc8c24fc908e7e2ebcb"}
Jul 02 14:36:55.844914 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:36:55.844881    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:37:01.794927 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:01.794889    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bkmqr/must-gather-9thgl" event={"ID":"65dae769-a7e6-454a-bb17-beed3dfc2caf","Type":"ContainerStarted","Data":"0648c122d651e339e511936e7a436623dcebe619cbb51953e6b1ea93401d5255"}
Jul 02 14:37:02.803588 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:02.803547    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bkmqr/must-gather-9thgl" event={"ID":"65dae769-a7e6-454a-bb17-beed3dfc2caf","Type":"ContainerStarted","Data":"5c8c331c512b5af24e13be1bab00be19141f39712ed0af0073f3db21f097caac"}
Jul 02 14:37:02.820632 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:02.820580    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bkmqr/must-gather-9thgl" podStartSLOduration=2.068816425 podStartE2EDuration="8.820562283s" totalImagesPullingTime="6.751745858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:36:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:37:02.819730318 +0000 UTC m=+4934.047641492" watchObservedRunningTime="2026-07-02 14:37:02.820562283 +0000 UTC m=+4934.048473457"
Jul 02 14:37:11.273910 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:11.273827    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-9pg7p_970f3494-f719-471d-818d-04f90719f2a3/storage-initializer/0.log"
Jul 02 14:37:11.415270 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:11.415242    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t_efdc86a8-bc40-445c-bd40-d607151733b7/main/0.log"
Jul 02 14:37:11.477193 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:11.477158    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-gg6n8_91bed416-a9bf-4dcf-af67-bceba2ece5ff/istio-proxy/0.log"
Jul 02 14:37:11.479117 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:11.479097    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-gg6n8_91bed416-a9bf-4dcf-af67-bceba2ece5ff/istio-proxy/1.log"
Jul 02 14:37:11.494847 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:11.494811    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-snkw7_1875f4df-a74b-4235-8bfd-c2e6b8aa64aa/istio-proxy/1.log"
Jul 02 14:37:11.495706 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:11.495685    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-snkw7_1875f4df-a74b-4235-8bfd-c2e6b8aa64aa/istio-proxy/0.log"
Jul 02 14:37:12.973865 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:12.973809    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-9pg7p_970f3494-f719-471d-818d-04f90719f2a3/storage-initializer/0.log"
Jul 02 14:37:13.081133 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:13.081102    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t_efdc86a8-bc40-445c-bd40-d607151733b7/main/0.log"
Jul 02 14:37:13.119773 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:13.119735    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-gg6n8_91bed416-a9bf-4dcf-af67-bceba2ece5ff/istio-proxy/0.log"
Jul 02 14:37:13.122433 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:13.122409    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-gg6n8_91bed416-a9bf-4dcf-af67-bceba2ece5ff/istio-proxy/1.log"
Jul 02 14:37:13.138715 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:13.138686    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-snkw7_1875f4df-a74b-4235-8bfd-c2e6b8aa64aa/istio-proxy/1.log"
Jul 02 14:37:13.139093 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:13.139061    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-snkw7_1875f4df-a74b-4235-8bfd-c2e6b8aa64aa/istio-proxy/0.log"
Jul 02 14:37:14.622419 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:14.622359    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-9pg7p_970f3494-f719-471d-818d-04f90719f2a3/storage-initializer/0.log"
Jul 02 14:37:14.720996 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:14.720962    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t_efdc86a8-bc40-445c-bd40-d607151733b7/main/0.log"
Jul 02 14:37:14.752990 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:14.752955    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-gg6n8_91bed416-a9bf-4dcf-af67-bceba2ece5ff/istio-proxy/0.log"
Jul 02 14:37:14.755808 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:14.755769    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-gg6n8_91bed416-a9bf-4dcf-af67-bceba2ece5ff/istio-proxy/1.log"
Jul 02 14:37:14.769734 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:14.769707    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-snkw7_1875f4df-a74b-4235-8bfd-c2e6b8aa64aa/istio-proxy/1.log"
Jul 02 14:37:14.770111 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:14.770089    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-snkw7_1875f4df-a74b-4235-8bfd-c2e6b8aa64aa/istio-proxy/0.log"
Jul 02 14:37:16.249842 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:16.249789    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-9pg7p_970f3494-f719-471d-818d-04f90719f2a3/storage-initializer/0.log"
Jul 02 14:37:16.349278 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:16.349239    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t_efdc86a8-bc40-445c-bd40-d607151733b7/main/0.log"
Jul 02 14:37:16.382826 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:16.382783    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-gg6n8_91bed416-a9bf-4dcf-af67-bceba2ece5ff/istio-proxy/0.log"
Jul 02 14:37:16.385266 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:16.385236    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-gg6n8_91bed416-a9bf-4dcf-af67-bceba2ece5ff/istio-proxy/1.log"
Jul 02 14:37:16.399647 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:16.399616    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-snkw7_1875f4df-a74b-4235-8bfd-c2e6b8aa64aa/istio-proxy/0.log"
Jul 02 14:37:16.399913 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:16.399888    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-snkw7_1875f4df-a74b-4235-8bfd-c2e6b8aa64aa/istio-proxy/1.log"
Jul 02 14:37:17.915341 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:17.915284    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-9pg7p_970f3494-f719-471d-818d-04f90719f2a3/storage-initializer/0.log"
Jul 02 14:37:18.008711 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:18.008675    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t_efdc86a8-bc40-445c-bd40-d607151733b7/main/0.log"
Jul 02 14:37:18.040276 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:18.040241    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-gg6n8_91bed416-a9bf-4dcf-af67-bceba2ece5ff/istio-proxy/0.log"
Jul 02 14:37:18.042950 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:18.042921    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-gg6n8_91bed416-a9bf-4dcf-af67-bceba2ece5ff/istio-proxy/1.log"
Jul 02 14:37:18.059757 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:18.059712    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-snkw7_1875f4df-a74b-4235-8bfd-c2e6b8aa64aa/istio-proxy/1.log"
Jul 02 14:37:18.060211 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:18.060190    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-snkw7_1875f4df-a74b-4235-8bfd-c2e6b8aa64aa/istio-proxy/0.log"
Jul 02 14:37:19.485898 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:19.485866    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-9pg7p_970f3494-f719-471d-818d-04f90719f2a3/storage-initializer/0.log"
Jul 02 14:37:19.581221 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:19.581188    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t_efdc86a8-bc40-445c-bd40-d607151733b7/main/0.log"
Jul 02 14:37:19.620356 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:19.620309    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-gg6n8_91bed416-a9bf-4dcf-af67-bceba2ece5ff/istio-proxy/0.log"
Jul 02 14:37:19.623034 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:19.623010    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-gg6n8_91bed416-a9bf-4dcf-af67-bceba2ece5ff/istio-proxy/1.log"
Jul 02 14:37:19.638098 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:19.638067    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-snkw7_1875f4df-a74b-4235-8bfd-c2e6b8aa64aa/istio-proxy/1.log"
Jul 02 14:37:19.638731 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:19.638706    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-snkw7_1875f4df-a74b-4235-8bfd-c2e6b8aa64aa/istio-proxy/0.log"
Jul 02 14:37:21.171828 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:21.171794    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-9pg7p_970f3494-f719-471d-818d-04f90719f2a3/storage-initializer/0.log"
Jul 02 14:37:21.284818 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:21.284778    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t_efdc86a8-bc40-445c-bd40-d607151733b7/main/0.log"
Jul 02 14:37:21.318263 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:21.318234    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-gg6n8_91bed416-a9bf-4dcf-af67-bceba2ece5ff/istio-proxy/0.log"
Jul 02 14:37:21.321065 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:21.321039    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-gg6n8_91bed416-a9bf-4dcf-af67-bceba2ece5ff/istio-proxy/1.log"
Jul 02 14:37:21.335708 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:21.335679    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-snkw7_1875f4df-a74b-4235-8bfd-c2e6b8aa64aa/istio-proxy/1.log"
Jul 02 14:37:21.336207 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:21.336179    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-snkw7_1875f4df-a74b-4235-8bfd-c2e6b8aa64aa/istio-proxy/0.log"
Jul 02 14:37:23.107634 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:23.107596    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-9pg7p_970f3494-f719-471d-818d-04f90719f2a3/storage-initializer/0.log"
Jul 02 14:37:23.246218 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:23.246186    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t_efdc86a8-bc40-445c-bd40-d607151733b7/main/0.log"
Jul 02 14:37:23.283870 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:23.283841    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-gg6n8_91bed416-a9bf-4dcf-af67-bceba2ece5ff/istio-proxy/0.log"
Jul 02 14:37:23.286620 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:23.286593    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-gg6n8_91bed416-a9bf-4dcf-af67-bceba2ece5ff/istio-proxy/1.log"
Jul 02 14:37:23.300371 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:23.300339    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-snkw7_1875f4df-a74b-4235-8bfd-c2e6b8aa64aa/istio-proxy/1.log"
Jul 02 14:37:23.300875 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:23.300847    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-snkw7_1875f4df-a74b-4235-8bfd-c2e6b8aa64aa/istio-proxy/0.log"
Jul 02 14:37:24.777771 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:24.777740    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-9pg7p_970f3494-f719-471d-818d-04f90719f2a3/storage-initializer/0.log"
Jul 02 14:37:24.875017 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:24.874980    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t_efdc86a8-bc40-445c-bd40-d607151733b7/main/0.log"
Jul 02 14:37:24.908513 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:24.908277    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-gg6n8_91bed416-a9bf-4dcf-af67-bceba2ece5ff/istio-proxy/0.log"
Jul 02 14:37:24.911570 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:24.911543    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-gg6n8_91bed416-a9bf-4dcf-af67-bceba2ece5ff/istio-proxy/1.log"
Jul 02 14:37:24.926157 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:24.926098    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-snkw7_1875f4df-a74b-4235-8bfd-c2e6b8aa64aa/istio-proxy/1.log"
Jul 02 14:37:24.926616 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:24.926596    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-snkw7_1875f4df-a74b-4235-8bfd-c2e6b8aa64aa/istio-proxy/0.log"
Jul 02 14:37:26.425195 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:26.425161    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-9pg7p_970f3494-f719-471d-818d-04f90719f2a3/storage-initializer/0.log"
Jul 02 14:37:26.535461 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:26.535432    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t_efdc86a8-bc40-445c-bd40-d607151733b7/main/0.log"
Jul 02 14:37:26.570374 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:26.570335    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-gg6n8_91bed416-a9bf-4dcf-af67-bceba2ece5ff/istio-proxy/0.log"
Jul 02 14:37:26.573018 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:26.572992    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-gg6n8_91bed416-a9bf-4dcf-af67-bceba2ece5ff/istio-proxy/1.log"
Jul 02 14:37:26.587818 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:26.587789    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-snkw7_1875f4df-a74b-4235-8bfd-c2e6b8aa64aa/istio-proxy/1.log"
Jul 02 14:37:26.588303 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:26.588283    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-snkw7_1875f4df-a74b-4235-8bfd-c2e6b8aa64aa/istio-proxy/0.log"
Jul 02 14:37:28.019644 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:28.019615    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-9pg7p_970f3494-f719-471d-818d-04f90719f2a3/storage-initializer/0.log"
Jul 02 14:37:28.115532 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:28.115496    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t_efdc86a8-bc40-445c-bd40-d607151733b7/main/0.log"
Jul 02 14:37:28.155021 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:28.154953    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-gg6n8_91bed416-a9bf-4dcf-af67-bceba2ece5ff/istio-proxy/0.log"
Jul 02 14:37:28.157706 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:28.157677    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-gg6n8_91bed416-a9bf-4dcf-af67-bceba2ece5ff/istio-proxy/1.log"
Jul 02 14:37:28.171847 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:28.171820    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-snkw7_1875f4df-a74b-4235-8bfd-c2e6b8aa64aa/istio-proxy/1.log"
Jul 02 14:37:28.172285 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:28.172265    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-snkw7_1875f4df-a74b-4235-8bfd-c2e6b8aa64aa/istio-proxy/0.log"
Jul 02 14:37:29.632715 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:29.632667    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-9pg7p_970f3494-f719-471d-818d-04f90719f2a3/storage-initializer/0.log"
Jul 02 14:37:29.725381 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:29.725346    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t_efdc86a8-bc40-445c-bd40-d607151733b7/main/0.log"
Jul 02 14:37:29.760217 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:29.760184    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-gg6n8_91bed416-a9bf-4dcf-af67-bceba2ece5ff/istio-proxy/0.log"
Jul 02 14:37:29.762684 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:29.762655    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-gg6n8_91bed416-a9bf-4dcf-af67-bceba2ece5ff/istio-proxy/1.log"
Jul 02 14:37:29.776698 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:29.776672    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-snkw7_1875f4df-a74b-4235-8bfd-c2e6b8aa64aa/istio-proxy/1.log"
Jul 02 14:37:29.777321 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:29.777294    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-snkw7_1875f4df-a74b-4235-8bfd-c2e6b8aa64aa/istio-proxy/0.log"
Jul 02 14:37:31.237001 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:31.236971    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-9pg7p_970f3494-f719-471d-818d-04f90719f2a3/storage-initializer/0.log"
Jul 02 14:37:31.330449 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:31.330422    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t_efdc86a8-bc40-445c-bd40-d607151733b7/main/0.log"
Jul 02 14:37:31.366154 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:31.366119    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-gg6n8_91bed416-a9bf-4dcf-af67-bceba2ece5ff/istio-proxy/0.log"
Jul 02 14:37:31.368911 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:31.368886    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-gg6n8_91bed416-a9bf-4dcf-af67-bceba2ece5ff/istio-proxy/1.log"
Jul 02 14:37:31.382859 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:31.382829    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-snkw7_1875f4df-a74b-4235-8bfd-c2e6b8aa64aa/istio-proxy/1.log"
Jul 02 14:37:31.383298 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:31.383276    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-snkw7_1875f4df-a74b-4235-8bfd-c2e6b8aa64aa/istio-proxy/0.log"
Jul 02 14:37:32.877216 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:32.877184    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-9pg7p_970f3494-f719-471d-818d-04f90719f2a3/storage-initializer/0.log"
Jul 02 14:37:32.965496 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:32.965459    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t_efdc86a8-bc40-445c-bd40-d607151733b7/main/0.log"
Jul 02 14:37:32.998040 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:32.998006    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-gg6n8_91bed416-a9bf-4dcf-af67-bceba2ece5ff/istio-proxy/0.log"
Jul 02 14:37:33.000686 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:33.000661    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-gg6n8_91bed416-a9bf-4dcf-af67-bceba2ece5ff/istio-proxy/1.log"
Jul 02 14:37:33.014137 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:33.014104    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-snkw7_1875f4df-a74b-4235-8bfd-c2e6b8aa64aa/istio-proxy/1.log"
Jul 02 14:37:33.014787 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:33.014766    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-snkw7_1875f4df-a74b-4235-8bfd-c2e6b8aa64aa/istio-proxy/0.log"
Jul 02 14:37:34.471970 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:34.471939    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-9pg7p_970f3494-f719-471d-818d-04f90719f2a3/storage-initializer/0.log"
Jul 02 14:37:34.566669 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:34.566643    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t_efdc86a8-bc40-445c-bd40-d607151733b7/main/0.log"
Jul 02 14:37:34.601717 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:34.601683    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-gg6n8_91bed416-a9bf-4dcf-af67-bceba2ece5ff/istio-proxy/0.log"
Jul 02 14:37:34.604365 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:34.604341    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-gg6n8_91bed416-a9bf-4dcf-af67-bceba2ece5ff/istio-proxy/1.log"
Jul 02 14:37:34.618655 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:34.618613    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-snkw7_1875f4df-a74b-4235-8bfd-c2e6b8aa64aa/istio-proxy/1.log"
Jul 02 14:37:34.619051 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:34.619034    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-snkw7_1875f4df-a74b-4235-8bfd-c2e6b8aa64aa/istio-proxy/0.log"
Jul 02 14:37:36.092905 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:36.092876    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-9pg7p_970f3494-f719-471d-818d-04f90719f2a3/storage-initializer/0.log"
Jul 02 14:37:36.186946 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:36.186905    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schetmw9t_efdc86a8-bc40-445c-bd40-d607151733b7/main/0.log"
Jul 02 14:37:36.219122 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:36.219089    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-gg6n8_91bed416-a9bf-4dcf-af67-bceba2ece5ff/istio-proxy/0.log"
Jul 02 14:37:36.221675 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:36.221651    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-gg6n8_91bed416-a9bf-4dcf-af67-bceba2ece5ff/istio-proxy/1.log"
Jul 02 14:37:36.235193 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:36.235168    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-snkw7_1875f4df-a74b-4235-8bfd-c2e6b8aa64aa/istio-proxy/1.log"
Jul 02 14:37:36.235690 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:36.235589    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-snkw7_1875f4df-a74b-4235-8bfd-c2e6b8aa64aa/istio-proxy/0.log"
Jul 02 14:37:44.141180 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:44.141141    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zf5kz_5857b5b0-7614-43f3-bc37-f6f6d88d0880/authorino/0.log"
Jul 02 14:37:44.156920 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:44.156891    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-h7pt5_0b801638-1cfe-4e8f-a194-d5f2a8c714e2/manager/0.log"
Jul 02 14:37:45.495460 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:45.495424    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zf5kz_5857b5b0-7614-43f3-bc37-f6f6d88d0880/authorino/0.log"
Jul 02 14:37:45.511893 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:45.511849    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-h7pt5_0b801638-1cfe-4e8f-a194-d5f2a8c714e2/manager/0.log"
Jul 02 14:37:46.886912 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:46.886873    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zf5kz_5857b5b0-7614-43f3-bc37-f6f6d88d0880/authorino/0.log"
Jul 02 14:37:46.905752 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:46.905709    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-h7pt5_0b801638-1cfe-4e8f-a194-d5f2a8c714e2/manager/0.log"
Jul 02 14:37:48.295752 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:48.295721    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zf5kz_5857b5b0-7614-43f3-bc37-f6f6d88d0880/authorino/0.log"
Jul 02 14:37:48.310312 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:48.310279    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-h7pt5_0b801638-1cfe-4e8f-a194-d5f2a8c714e2/manager/0.log"
Jul 02 14:37:49.650916 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:49.650857    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zf5kz_5857b5b0-7614-43f3-bc37-f6f6d88d0880/authorino/0.log"
Jul 02 14:37:49.669053 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:49.669019    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-h7pt5_0b801638-1cfe-4e8f-a194-d5f2a8c714e2/manager/0.log"
Jul 02 14:37:51.015109 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:51.015076    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zf5kz_5857b5b0-7614-43f3-bc37-f6f6d88d0880/authorino/0.log"
Jul 02 14:37:51.031522 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:51.031487    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-h7pt5_0b801638-1cfe-4e8f-a194-d5f2a8c714e2/manager/0.log"
Jul 02 14:37:52.384082 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:52.384049    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zf5kz_5857b5b0-7614-43f3-bc37-f6f6d88d0880/authorino/0.log"
Jul 02 14:37:52.402682 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:52.402650    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-h7pt5_0b801638-1cfe-4e8f-a194-d5f2a8c714e2/manager/0.log"
Jul 02 14:37:53.752115 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:53.752084    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zf5kz_5857b5b0-7614-43f3-bc37-f6f6d88d0880/authorino/0.log"
Jul 02 14:37:53.769283 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:53.769251    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-h7pt5_0b801638-1cfe-4e8f-a194-d5f2a8c714e2/manager/0.log"
Jul 02 14:37:55.103218 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:55.103184    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zf5kz_5857b5b0-7614-43f3-bc37-f6f6d88d0880/authorino/0.log"
Jul 02 14:37:55.119282 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:55.119246    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-h7pt5_0b801638-1cfe-4e8f-a194-d5f2a8c714e2/manager/0.log"
Jul 02 14:37:56.504711 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:56.504676    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zf5kz_5857b5b0-7614-43f3-bc37-f6f6d88d0880/authorino/0.log"
Jul 02 14:37:56.527423 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:56.527366    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-h7pt5_0b801638-1cfe-4e8f-a194-d5f2a8c714e2/manager/0.log"
Jul 02 14:37:57.913100 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:57.913061    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zf5kz_5857b5b0-7614-43f3-bc37-f6f6d88d0880/authorino/0.log"
Jul 02 14:37:57.931250 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:57.931215    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-h7pt5_0b801638-1cfe-4e8f-a194-d5f2a8c714e2/manager/0.log"
Jul 02 14:37:59.261056 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:59.261021    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zf5kz_5857b5b0-7614-43f3-bc37-f6f6d88d0880/authorino/0.log"
Jul 02 14:37:59.280172 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:37:59.280140    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-h7pt5_0b801638-1cfe-4e8f-a194-d5f2a8c714e2/manager/0.log"
Jul 02 14:38:00.604499 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:00.604451    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zf5kz_5857b5b0-7614-43f3-bc37-f6f6d88d0880/authorino/0.log"
Jul 02 14:38:00.631935 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:00.631905    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-h7pt5_0b801638-1cfe-4e8f-a194-d5f2a8c714e2/manager/0.log"
Jul 02 14:38:01.984200 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:01.984152    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zf5kz_5857b5b0-7614-43f3-bc37-f6f6d88d0880/authorino/0.log"
Jul 02 14:38:01.999509 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:01.999478    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-h7pt5_0b801638-1cfe-4e8f-a194-d5f2a8c714e2/manager/0.log"
Jul 02 14:38:03.335058 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:03.335023    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zf5kz_5857b5b0-7614-43f3-bc37-f6f6d88d0880/authorino/0.log"
Jul 02 14:38:03.352344 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:03.352314    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-h7pt5_0b801638-1cfe-4e8f-a194-d5f2a8c714e2/manager/0.log"
Jul 02 14:38:04.669217 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:04.669174    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zf5kz_5857b5b0-7614-43f3-bc37-f6f6d88d0880/authorino/0.log"
Jul 02 14:38:04.683583 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:04.683551    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-h7pt5_0b801638-1cfe-4e8f-a194-d5f2a8c714e2/manager/0.log"
Jul 02 14:38:06.014812 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:06.014771    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zf5kz_5857b5b0-7614-43f3-bc37-f6f6d88d0880/authorino/0.log"
Jul 02 14:38:06.032589 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:06.032557    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-h7pt5_0b801638-1cfe-4e8f-a194-d5f2a8c714e2/manager/0.log"
Jul 02 14:38:07.428583 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:07.428553    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zf5kz_5857b5b0-7614-43f3-bc37-f6f6d88d0880/authorino/0.log"
Jul 02 14:38:07.449024 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:07.448993    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-h7pt5_0b801638-1cfe-4e8f-a194-d5f2a8c714e2/manager/0.log"
Jul 02 14:38:08.850055 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:08.850017    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zf5kz_5857b5b0-7614-43f3-bc37-f6f6d88d0880/authorino/0.log"
Jul 02 14:38:08.866151 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:08.866109    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-h7pt5_0b801638-1cfe-4e8f-a194-d5f2a8c714e2/manager/0.log"
Jul 02 14:38:10.215298 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:10.215269    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zf5kz_5857b5b0-7614-43f3-bc37-f6f6d88d0880/authorino/0.log"
Jul 02 14:38:10.229852 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:10.229816    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-h7pt5_0b801638-1cfe-4e8f-a194-d5f2a8c714e2/manager/0.log"
Jul 02 14:38:11.610277 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:11.610225    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zf5kz_5857b5b0-7614-43f3-bc37-f6f6d88d0880/authorino/0.log"
Jul 02 14:38:11.644853 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:11.644821    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-h7pt5_0b801638-1cfe-4e8f-a194-d5f2a8c714e2/manager/0.log"
Jul 02 14:38:13.186611 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:13.186574    2583 generic.go:356] "Generic (PLEG): container finished" podID="65dae769-a7e6-454a-bb17-beed3dfc2caf" containerID="0648c122d651e339e511936e7a436623dcebe619cbb51953e6b1ea93401d5255" exitCode=0
Jul 02 14:38:13.187044 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:13.186650    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bkmqr/must-gather-9thgl" event={"ID":"65dae769-a7e6-454a-bb17-beed3dfc2caf","Type":"ContainerDied","Data":"0648c122d651e339e511936e7a436623dcebe619cbb51953e6b1ea93401d5255"}
Jul 02 14:38:13.187092 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:13.187039    2583 scope.go:117] "RemoveContainer" containerID="0648c122d651e339e511936e7a436623dcebe619cbb51953e6b1ea93401d5255"
Jul 02 14:38:13.760553 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:13.760522    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bkmqr_must-gather-9thgl_65dae769-a7e6-454a-bb17-beed3dfc2caf/gather/0.log"
Jul 02 14:38:14.416886 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:14.416852    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:38:14.549121 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:14.549086    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zfr7l/must-gather-x7zxr"]
Jul 02 14:38:14.553800 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:14.553782    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zfr7l/must-gather-x7zxr"
Jul 02 14:38:14.556470 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:14.556448    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zfr7l\"/\"openshift-service-ca.crt\""
Jul 02 14:38:14.557430 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:14.557413    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zfr7l\"/\"kube-root-ca.crt\""
Jul 02 14:38:14.557523 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:14.557501    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zfr7l\"/\"default-dockercfg-cvgg4\""
Jul 02 14:38:14.562251 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:14.562220    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:38:14.562967 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:14.562947    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zfr7l/must-gather-x7zxr"]
Jul 02 14:38:14.653064 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:14.653027    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl2mm\" (UniqueName: \"kubernetes.io/projected/f8196f44-ccf3-4ec3-8846-28cc63cb0423-kube-api-access-nl2mm\") pod \"must-gather-x7zxr\" (UID: \"f8196f44-ccf3-4ec3-8846-28cc63cb0423\") " pod="openshift-must-gather-zfr7l/must-gather-x7zxr"
Jul 02 14:38:14.653246 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:14.653071    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f8196f44-ccf3-4ec3-8846-28cc63cb0423-must-gather-output\") pod \"must-gather-x7zxr\" (UID: \"f8196f44-ccf3-4ec3-8846-28cc63cb0423\") " pod="openshift-must-gather-zfr7l/must-gather-x7zxr"
Jul 02 14:38:14.658409 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:14.658365    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:38:14.754426 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:14.754308    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl2mm\" (UniqueName: \"kubernetes.io/projected/f8196f44-ccf3-4ec3-8846-28cc63cb0423-kube-api-access-nl2mm\") pod \"must-gather-x7zxr\" (UID: \"f8196f44-ccf3-4ec3-8846-28cc63cb0423\") " pod="openshift-must-gather-zfr7l/must-gather-x7zxr"
Jul 02 14:38:14.754426 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:14.754355    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f8196f44-ccf3-4ec3-8846-28cc63cb0423-must-gather-output\") pod \"must-gather-x7zxr\" (UID: \"f8196f44-ccf3-4ec3-8846-28cc63cb0423\") " pod="openshift-must-gather-zfr7l/must-gather-x7zxr"
Jul 02 14:38:14.754694 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:14.754675    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f8196f44-ccf3-4ec3-8846-28cc63cb0423-must-gather-output\") pod \"must-gather-x7zxr\" (UID: \"f8196f44-ccf3-4ec3-8846-28cc63cb0423\") " pod="openshift-must-gather-zfr7l/must-gather-x7zxr"
Jul 02 14:38:14.763207 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:14.763179    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl2mm\" (UniqueName: \"kubernetes.io/projected/f8196f44-ccf3-4ec3-8846-28cc63cb0423-kube-api-access-nl2mm\") pod \"must-gather-x7zxr\" (UID: \"f8196f44-ccf3-4ec3-8846-28cc63cb0423\") " pod="openshift-must-gather-zfr7l/must-gather-x7zxr"
Jul 02 14:38:14.864510 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:14.864467    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zfr7l/must-gather-x7zxr"
Jul 02 14:38:14.999936 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:14.999902    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zfr7l/must-gather-x7zxr"]
Jul 02 14:38:15.003281 ip-10-0-135-143 kubenswrapper[2583]: W0702 14:38:15.003253    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8196f44_ccf3_4ec3_8846_28cc63cb0423.slice/crio-ab32316610776fd27e0f343d350526b7588057b5d9e41f53470a4497a8f2fe80 WatchSource:0}: Error finding container ab32316610776fd27e0f343d350526b7588057b5d9e41f53470a4497a8f2fe80: Status 404 returned error can't find the container with id ab32316610776fd27e0f343d350526b7588057b5d9e41f53470a4497a8f2fe80
Jul 02 14:38:15.197682 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:15.197647    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zfr7l/must-gather-x7zxr" event={"ID":"f8196f44-ccf3-4ec3-8846-28cc63cb0423","Type":"ContainerStarted","Data":"ab32316610776fd27e0f343d350526b7588057b5d9e41f53470a4497a8f2fe80"}
Jul 02 14:38:16.207377 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:16.207309    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zfr7l/must-gather-x7zxr" event={"ID":"f8196f44-ccf3-4ec3-8846-28cc63cb0423","Type":"ContainerStarted","Data":"5e650630426112d0d92c1d6714fcf2a99869a76b3261911b6a36c065f0d0e89c"}
Jul 02 14:38:17.216264 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:17.216221    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zfr7l/must-gather-x7zxr" event={"ID":"f8196f44-ccf3-4ec3-8846-28cc63cb0423","Type":"ContainerStarted","Data":"681d1b63696027569104847f2bbeee0914d6585ee5e1ca50dce92769e965c9aa"}
Jul 02 14:38:17.238354 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:17.238280    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zfr7l/must-gather-x7zxr" podStartSLOduration=2.1605771320000002 podStartE2EDuration="3.238259494s" totalImagesPullingTime="1.077682362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:38:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:38:17.234516443 +0000 UTC m=+5008.462427617" watchObservedRunningTime="2026-07-02 14:38:17.238259494 +0000 UTC m=+5008.466170685"
Jul 02 14:38:17.697461 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:17.697429    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9fpt5_bfa11a9f-ca92-4214-8e84-0b9c48a8337c/global-pull-secret-syncer/0.log"
Jul 02 14:38:17.832046 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:17.832010    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log"
Jul 02 14:38:17.994287 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:17.994202    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-143.ec2.internal_b72efbb5a6c47d6a170c394c0a860619/haproxy/0.log"
Jul 02 14:38:19.402125 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:19.402082    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-bkmqr/must-gather-9thgl"]
Jul 02 14:38:19.402728 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:19.402315    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-bkmqr/must-gather-9thgl" podUID="65dae769-a7e6-454a-bb17-beed3dfc2caf" containerName="copy" containerID="cri-o://5c8c331c512b5af24e13be1bab00be19141f39712ed0af0073f3db21f097caac" gracePeriod=2
Jul 02 14:38:19.405250 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:38:19.405216    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-9thgl\" is forbidden: User \"system:node:ip-10-0-135-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bkmqr\": no relationship found between node 'ip-10-0-135-143.ec2.internal' and this object" podUID="65dae769-a7e6-454a-bb17-beed3dfc2caf" pod="openshift-must-gather-bkmqr/must-gather-9thgl"
Jul 02 14:38:19.407484 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:19.407462    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-bkmqr/must-gather-9thgl"]
Jul 02 14:38:19.779120 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:19.776722    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bkmqr_must-gather-9thgl_65dae769-a7e6-454a-bb17-beed3dfc2caf/copy/0.log"
Jul 02 14:38:19.788690 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:19.786365    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bkmqr/must-gather-9thgl"
Jul 02 14:38:19.788859 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:38:19.788738    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-9thgl\" is forbidden: User \"system:node:ip-10-0-135-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bkmqr\": no relationship found between node 'ip-10-0-135-143.ec2.internal' and this object" podUID="65dae769-a7e6-454a-bb17-beed3dfc2caf" pod="openshift-must-gather-bkmqr/must-gather-9thgl"
Jul 02 14:38:19.816413 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:19.814648    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9k5xs\" (UniqueName: \"kubernetes.io/projected/65dae769-a7e6-454a-bb17-beed3dfc2caf-kube-api-access-9k5xs\") pod \"65dae769-a7e6-454a-bb17-beed3dfc2caf\" (UID: \"65dae769-a7e6-454a-bb17-beed3dfc2caf\") "
Jul 02 14:38:19.816413 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:19.814720    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/65dae769-a7e6-454a-bb17-beed3dfc2caf-must-gather-output\") pod \"65dae769-a7e6-454a-bb17-beed3dfc2caf\" (UID: \"65dae769-a7e6-454a-bb17-beed3dfc2caf\") "
Jul 02 14:38:19.827472 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:19.825196    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65dae769-a7e6-454a-bb17-beed3dfc2caf-kube-api-access-9k5xs" (OuterVolumeSpecName: "kube-api-access-9k5xs") pod "65dae769-a7e6-454a-bb17-beed3dfc2caf" (UID: "65dae769-a7e6-454a-bb17-beed3dfc2caf"). InnerVolumeSpecName "kube-api-access-9k5xs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:38:19.827472 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:19.827130    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65dae769-a7e6-454a-bb17-beed3dfc2caf-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "65dae769-a7e6-454a-bb17-beed3dfc2caf" (UID: "65dae769-a7e6-454a-bb17-beed3dfc2caf"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:38:19.916142 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:19.916097    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k5xs\" (UniqueName: \"kubernetes.io/projected/65dae769-a7e6-454a-bb17-beed3dfc2caf-kube-api-access-9k5xs\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:38:19.916142 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:19.916140    2583 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/65dae769-a7e6-454a-bb17-beed3dfc2caf-must-gather-output\") on node \"ip-10-0-135-143.ec2.internal\" DevicePath \"\""
Jul 02 14:38:20.240640 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:20.240598    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-bkmqr_must-gather-9thgl_65dae769-a7e6-454a-bb17-beed3dfc2caf/copy/0.log"
Jul 02 14:38:20.241067 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:20.241027    2583 generic.go:356] "Generic (PLEG): container finished" podID="65dae769-a7e6-454a-bb17-beed3dfc2caf" containerID="5c8c331c512b5af24e13be1bab00be19141f39712ed0af0073f3db21f097caac" exitCode=143
Jul 02 14:38:20.241184 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:20.241104    2583 scope.go:117] "RemoveContainer" containerID="5c8c331c512b5af24e13be1bab00be19141f39712ed0af0073f3db21f097caac"
Jul 02 14:38:20.241319 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:20.241282    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bkmqr/must-gather-9thgl"
Jul 02 14:38:20.246526 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:38:20.245050    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-9thgl\" is forbidden: User \"system:node:ip-10-0-135-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bkmqr\": no relationship found between node 'ip-10-0-135-143.ec2.internal' and this object" podUID="65dae769-a7e6-454a-bb17-beed3dfc2caf" pod="openshift-must-gather-bkmqr/must-gather-9thgl"
Jul 02 14:38:20.255683 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:20.255642    2583 scope.go:117] "RemoveContainer" containerID="0648c122d651e339e511936e7a436623dcebe619cbb51953e6b1ea93401d5255"
Jul 02 14:38:20.270050 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:38:20.264639    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-9thgl\" is forbidden: User \"system:node:ip-10-0-135-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-bkmqr\": no relationship found between node 'ip-10-0-135-143.ec2.internal' and this object" podUID="65dae769-a7e6-454a-bb17-beed3dfc2caf" pod="openshift-must-gather-bkmqr/must-gather-9thgl"
Jul 02 14:38:20.297527 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:20.293727    2583 scope.go:117] "RemoveContainer" containerID="5c8c331c512b5af24e13be1bab00be19141f39712ed0af0073f3db21f097caac"
Jul 02 14:38:20.297527 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:38:20.294182    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c8c331c512b5af24e13be1bab00be19141f39712ed0af0073f3db21f097caac\": container with ID starting with 5c8c331c512b5af24e13be1bab00be19141f39712ed0af0073f3db21f097caac not found: ID does not exist" containerID="5c8c331c512b5af24e13be1bab00be19141f39712ed0af0073f3db21f097caac"
Jul 02 14:38:20.297527 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:20.294220    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c8c331c512b5af24e13be1bab00be19141f39712ed0af0073f3db21f097caac"} err="failed to get container status \"5c8c331c512b5af24e13be1bab00be19141f39712ed0af0073f3db21f097caac\": rpc error: code = NotFound desc = could not find container \"5c8c331c512b5af24e13be1bab00be19141f39712ed0af0073f3db21f097caac\": container with ID starting with 5c8c331c512b5af24e13be1bab00be19141f39712ed0af0073f3db21f097caac not found: ID does not exist"
Jul 02 14:38:20.297527 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:20.294249    2583 scope.go:117] "RemoveContainer" containerID="0648c122d651e339e511936e7a436623dcebe619cbb51953e6b1ea93401d5255"
Jul 02 14:38:20.297527 ip-10-0-135-143 kubenswrapper[2583]: E0702 14:38:20.294595    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0648c122d651e339e511936e7a436623dcebe619cbb51953e6b1ea93401d5255\": container with ID starting with 0648c122d651e339e511936e7a436623dcebe619cbb51953e6b1ea93401d5255 not found: ID does not exist" containerID="0648c122d651e339e511936e7a436623dcebe619cbb51953e6b1ea93401d5255"
Jul 02 14:38:20.297527 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:20.294625    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0648c122d651e339e511936e7a436623dcebe619cbb51953e6b1ea93401d5255"} err="failed to get container status \"0648c122d651e339e511936e7a436623dcebe619cbb51953e6b1ea93401d5255\": rpc error: code = NotFound desc = could not find container \"0648c122d651e339e511936e7a436623dcebe619cbb51953e6b1ea93401d5255\": container with ID starting with 0648c122d651e339e511936e7a436623dcebe619cbb51953e6b1ea93401d5255 not found: ID does not exist"
Jul 02 14:38:20.965495 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:20.965459    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:38:21.032666 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:21.032617    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7jfsq_4c5abfcf-f610-4f74-9222-a321d08c0a1b/konnectivity-agent/0.log" limit=1024
Jul 02 14:38:21.239405 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:21.239298    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65dae769-a7e6-454a-bb17-beed3dfc2caf" path="/var/lib/kubelet/pods/65dae769-a7e6-454a-bb17-beed3dfc2caf/volumes"
Jul 02 14:38:23.855041 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:23.854965    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-zf5kz_5857b5b0-7614-43f3-bc37-f6f6d88d0880/authorino/0.log"
Jul 02 14:38:23.923984 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:23.917659    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-h7pt5_0b801638-1cfe-4e8f-a194-d5f2a8c714e2/manager/0.log"
Jul 02 14:38:25.431123 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:25.431098    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a6c204f-4d22-4e76-982e-98dcb22daa59/alertmanager/0.log"
Jul 02 14:38:25.467875 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:25.467842    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a6c204f-4d22-4e76-982e-98dcb22daa59/config-reloader/0.log"
Jul 02 14:38:25.497453 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:25.497417    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a6c204f-4d22-4e76-982e-98dcb22daa59/kube-rbac-proxy-web/0.log"
Jul 02 14:38:25.523381 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:25.523332    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a6c204f-4d22-4e76-982e-98dcb22daa59/kube-rbac-proxy/0.log"
Jul 02 14:38:25.548905 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:25.548869    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a6c204f-4d22-4e76-982e-98dcb22daa59/kube-rbac-proxy-metric/0.log"
Jul 02 14:38:25.575576 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:25.575504    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a6c204f-4d22-4e76-982e-98dcb22daa59/prom-label-proxy/0.log"
Jul 02 14:38:25.616011 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:25.615976    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a6c204f-4d22-4e76-982e-98dcb22daa59/init-config-reloader/0.log"
Jul 02 14:38:25.714099 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:25.714011    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-drjzm_1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5/kube-state-metrics/0.log"
Jul 02 14:38:25.734164 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:25.734125    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-drjzm_1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5/kube-rbac-proxy-main/0.log"
Jul 02 14:38:25.756524 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:25.756489    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-drjzm_1f57e6e5-ce15-4e7f-9fcd-12f0de2546c5/kube-rbac-proxy-self/0.log"
Jul 02 14:38:25.784449 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:25.784418    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7557bdc5fb-gmwtd_886e2635-8f28-407b-99e6-cb8a3f869117/metrics-server/0.log"
Jul 02 14:38:25.816798 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:25.816762    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-cd8656b8-fq7jd_6e4c183e-20bc-408d-b975-bc6a9aafe76c/monitoring-plugin/0.log"
Jul 02 14:38:25.855517 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:25.855469    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pjp9n_a5e06a98-36dc-49b0-959c-d646b9802308/node-exporter/0.log"
Jul 02 14:38:25.882315 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:25.882281    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pjp9n_a5e06a98-36dc-49b0-959c-d646b9802308/kube-rbac-proxy/0.log"
Jul 02 14:38:25.904413 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:25.904356    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pjp9n_a5e06a98-36dc-49b0-959c-d646b9802308/init-textfile/0.log"
Jul 02 14:38:26.091453 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:26.091357    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-n9cz7_60c1f844-3734-4f58-9ed9-f0b9f370b096/kube-rbac-proxy-main/0.log"
Jul 02 14:38:26.115553 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:26.115512    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-n9cz7_60c1f844-3734-4f58-9ed9-f0b9f370b096/kube-rbac-proxy-self/0.log"
Jul 02 14:38:26.143948 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:26.143919    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-n9cz7_60c1f844-3734-4f58-9ed9-f0b9f370b096/openshift-state-metrics/0.log"
Jul 02 14:38:26.201899 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:26.201865    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_965caecb-d1eb-45a2-a48a-1495330553f8/prometheus/0.log"
Jul 02 14:38:26.222401 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:26.222352    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_965caecb-d1eb-45a2-a48a-1495330553f8/config-reloader/0.log"
Jul 02 14:38:26.248402 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:26.248369    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_965caecb-d1eb-45a2-a48a-1495330553f8/thanos-sidecar/0.log"
Jul 02 14:38:26.273258 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:26.273232    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_965caecb-d1eb-45a2-a48a-1495330553f8/kube-rbac-proxy-web/0.log"
Jul 02 14:38:26.301526 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:26.301500    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_965caecb-d1eb-45a2-a48a-1495330553f8/kube-rbac-proxy/0.log"
Jul 02 14:38:26.325532 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:26.325500    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_965caecb-d1eb-45a2-a48a-1495330553f8/kube-rbac-proxy-thanos/0.log"
Jul 02 14:38:26.350329 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:26.350242    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_965caecb-d1eb-45a2-a48a-1495330553f8/init-config-reloader/0.log"
Jul 02 14:38:26.383931 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:26.383896    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7b7d896497-78l79_330f7db3-f601-4139-9272-87bc867c9ad1/prometheus-operator/0.log"
Jul 02 14:38:26.408569 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:26.408519    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7b7d896497-78l79_330f7db3-f601-4139-9272-87bc867c9ad1/kube-rbac-proxy/0.log"
Jul 02 14:38:26.470242 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:26.470198    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76d8988fc8-tpstw_06e92769-f4b3-414a-b23c-ef41c5aa5af6/telemeter-client/0.log"
Jul 02 14:38:26.499060 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:26.499028    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76d8988fc8-tpstw_06e92769-f4b3-414a-b23c-ef41c5aa5af6/reload/0.log"
Jul 02 14:38:26.523501 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:26.523459    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76d8988fc8-tpstw_06e92769-f4b3-414a-b23c-ef41c5aa5af6/kube-rbac-proxy/0.log"
Jul 02 14:38:26.560434 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:26.560367    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57f4dddd4d-p6t8x_5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3/thanos-query/0.log"
Jul 02 14:38:26.583092 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:26.583057    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57f4dddd4d-p6t8x_5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3/kube-rbac-proxy-web/0.log"
Jul 02 14:38:26.604315 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:26.604229    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57f4dddd4d-p6t8x_5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3/kube-rbac-proxy/0.log"
Jul 02 14:38:26.629204 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:26.629175    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57f4dddd4d-p6t8x_5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3/prom-label-proxy/0.log"
Jul 02 14:38:26.653198 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:26.653165    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57f4dddd4d-p6t8x_5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3/kube-rbac-proxy-rules/0.log"
Jul 02 14:38:26.679540 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:26.679508    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57f4dddd4d-p6t8x_5c97d97e-b6a7-4f3f-80b8-e2dbd6857ba3/kube-rbac-proxy-metrics/0.log"
Jul 02 14:38:28.115415 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:28.115360    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zfr7l/perf-node-gather-daemonset-wx722"]
Jul 02 14:38:28.115998 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:28.115955    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65dae769-a7e6-454a-bb17-beed3dfc2caf" containerName="copy"
Jul 02 14:38:28.115998 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:28.115972    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="65dae769-a7e6-454a-bb17-beed3dfc2caf" containerName="copy"
Jul 02 14:38:28.116117 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:28.116009    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65dae769-a7e6-454a-bb17-beed3dfc2caf" containerName="gather"
Jul 02 14:38:28.116117 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:28.116018    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="65dae769-a7e6-454a-bb17-beed3dfc2caf" containerName="gather"
Jul 02 14:38:28.116117 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:28.116112    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="65dae769-a7e6-454a-bb17-beed3dfc2caf" containerName="gather"
Jul 02 14:38:28.116271 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:28.116126    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="65dae769-a7e6-454a-bb17-beed3dfc2caf" containerName="copy"
Jul 02 14:38:28.126562 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:28.126513    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-wx722"
Jul 02 14:38:28.133607 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:28.133569    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zfr7l/perf-node-gather-daemonset-wx722"]
Jul 02 14:38:28.216250 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:28.216200    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1b9adb0a-1746-4bfe-8efa-0d55431aa036-proc\") pod \"perf-node-gather-daemonset-wx722\" (UID: \"1b9adb0a-1746-4bfe-8efa-0d55431aa036\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-wx722"
Jul 02 14:38:28.216459 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:28.216276    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b9adb0a-1746-4bfe-8efa-0d55431aa036-lib-modules\") pod \"perf-node-gather-daemonset-wx722\" (UID: \"1b9adb0a-1746-4bfe-8efa-0d55431aa036\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-wx722"
Jul 02 14:38:28.216459 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:28.216308    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsxht\" (UniqueName: \"kubernetes.io/projected/1b9adb0a-1746-4bfe-8efa-0d55431aa036-kube-api-access-rsxht\") pod \"perf-node-gather-daemonset-wx722\" (UID: \"1b9adb0a-1746-4bfe-8efa-0d55431aa036\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-wx722"
Jul 02 14:38:28.216459 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:28.216339    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1b9adb0a-1746-4bfe-8efa-0d55431aa036-podres\") pod \"perf-node-gather-daemonset-wx722\" (UID: \"1b9adb0a-1746-4bfe-8efa-0d55431aa036\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-wx722"
Jul 02 14:38:28.216651 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:28.216470    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b9adb0a-1746-4bfe-8efa-0d55431aa036-sys\") pod \"perf-node-gather-daemonset-wx722\" (UID: \"1b9adb0a-1746-4bfe-8efa-0d55431aa036\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-wx722"
Jul 02 14:38:28.318026 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:28.317982    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b9adb0a-1746-4bfe-8efa-0d55431aa036-lib-modules\") pod \"perf-node-gather-daemonset-wx722\" (UID: \"1b9adb0a-1746-4bfe-8efa-0d55431aa036\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-wx722"
Jul 02 14:38:28.318216 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:28.318045    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsxht\" (UniqueName: \"kubernetes.io/projected/1b9adb0a-1746-4bfe-8efa-0d55431aa036-kube-api-access-rsxht\") pod \"perf-node-gather-daemonset-wx722\" (UID: \"1b9adb0a-1746-4bfe-8efa-0d55431aa036\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-wx722"
Jul 02 14:38:28.318216 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:28.318077    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1b9adb0a-1746-4bfe-8efa-0d55431aa036-podres\") pod \"perf-node-gather-daemonset-wx722\" (UID: \"1b9adb0a-1746-4bfe-8efa-0d55431aa036\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-wx722"
Jul 02 14:38:28.318216 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:28.318143    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b9adb0a-1746-4bfe-8efa-0d55431aa036-sys\") pod \"perf-node-gather-daemonset-wx722\" (UID: \"1b9adb0a-1746-4bfe-8efa-0d55431aa036\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-wx722"
Jul 02 14:38:28.318422 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:28.318266    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1b9adb0a-1746-4bfe-8efa-0d55431aa036-proc\") pod \"perf-node-gather-daemonset-wx722\" (UID: \"1b9adb0a-1746-4bfe-8efa-0d55431aa036\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-wx722"
Jul 02 14:38:28.318422 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:28.318365    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1b9adb0a-1746-4bfe-8efa-0d55431aa036-proc\") pod \"perf-node-gather-daemonset-wx722\" (UID: \"1b9adb0a-1746-4bfe-8efa-0d55431aa036\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-wx722"
Jul 02 14:38:28.318553 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:28.318540    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b9adb0a-1746-4bfe-8efa-0d55431aa036-lib-modules\") pod \"perf-node-gather-daemonset-wx722\" (UID: \"1b9adb0a-1746-4bfe-8efa-0d55431aa036\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-wx722"
Jul 02 14:38:28.318972 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:28.318941    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1b9adb0a-1746-4bfe-8efa-0d55431aa036-podres\") pod \"perf-node-gather-daemonset-wx722\" (UID: \"1b9adb0a-1746-4bfe-8efa-0d55431aa036\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-wx722"
Jul 02 14:38:28.319060 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:28.319013    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b9adb0a-1746-4bfe-8efa-0d55431aa036-sys\") pod \"perf-node-gather-daemonset-wx722\" (UID: \"1b9adb0a-1746-4bfe-8efa-0d55431aa036\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-wx722"
Jul 02 14:38:28.329725 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:28.329678    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsxht\" (UniqueName: \"kubernetes.io/projected/1b9adb0a-1746-4bfe-8efa-0d55431aa036-kube-api-access-rsxht\") pod \"perf-node-gather-daemonset-wx722\" (UID: \"1b9adb0a-1746-4bfe-8efa-0d55431aa036\") " pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-wx722"
Jul 02 14:38:28.444825 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:28.444736    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-wx722"
Jul 02 14:38:28.613966 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:28.613893    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zfr7l/perf-node-gather-daemonset-wx722"]
Jul 02 14:38:28.617673 ip-10-0-135-143 kubenswrapper[2583]: W0702 14:38:28.617637    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1b9adb0a_1746_4bfe_8efa_0d55431aa036.slice/crio-9d1770c7c2a6d58e3fd4450c09de53ecdd12548096cdb2964d67eacc3d8c4c91 WatchSource:0}: Error finding container 9d1770c7c2a6d58e3fd4450c09de53ecdd12548096cdb2964d67eacc3d8c4c91: Status 404 returned error can't find the container with id 9d1770c7c2a6d58e3fd4450c09de53ecdd12548096cdb2964d67eacc3d8c4c91
Jul 02 14:38:28.837020 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:28.836968    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5499b6fb8f-mxb5n_79b6d559-f1fe-45a4-9402-5c4008c2108b/download-server/0.log"
Jul 02 14:38:29.307967 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:29.307924    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-wx722" event={"ID":"1b9adb0a-1746-4bfe-8efa-0d55431aa036","Type":"ContainerStarted","Data":"6aab06ee702bb5ae57ada1a7b2fa255af29a7a377adc5295469e5cd4fc332e41"}
Jul 02 14:38:29.308476 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:29.307994    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-wx722" event={"ID":"1b9adb0a-1746-4bfe-8efa-0d55431aa036","Type":"ContainerStarted","Data":"9d1770c7c2a6d58e3fd4450c09de53ecdd12548096cdb2964d67eacc3d8c4c91"}
Jul 02 14:38:29.308476 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:29.308221    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-wx722"
Jul 02 14:38:30.623601 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:30.623566    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tkws4_74475ceb-1fd0-4b3d-a5f4-60c91249a748/dns/0.log"
Jul 02 14:38:30.649003 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:30.648976    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tkws4_74475ceb-1fd0-4b3d-a5f4-60c91249a748/kube-rbac-proxy/0.log"
Jul 02 14:38:30.742682 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:30.742650    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hzspc_a2b80dbb-5bb9-4d5e-add9-aeb0519f92c1/dns-node-resolver/0.log"
Jul 02 14:38:31.338579 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:31.338546    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9pvbh_b2cccfac-f483-4a64-8788-b78393b5a88e/node-ca/0.log"
Jul 02 14:38:33.543980 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:33.543949    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dd8n6_bc66f32c-20f9-44d2-b58e-5e3b8ede1958/serve-healthcheck-canary/0.log"
Jul 02 14:38:34.142039 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:34.141999    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n5cmj_3810cfce-45d5-47b2-950c-a17f109aa7ac/kube-rbac-proxy/0.log"
Jul 02 14:38:34.163713 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:34.163687    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n5cmj_3810cfce-45d5-47b2-950c-a17f109aa7ac/exporter/0.log"
Jul 02 14:38:34.186959 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:34.186927    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n5cmj_3810cfce-45d5-47b2-950c-a17f109aa7ac/extractor/0.log"
Jul 02 14:38:35.325307 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:35.325276    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-wx722"
Jul 02 14:38:35.344370 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:35.344316    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zfr7l/perf-node-gather-daemonset-wx722" podStartSLOduration=7.34430111 podStartE2EDuration="7.34430111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 14:38:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:38:29.328584017 +0000 UTC m=+5020.556495192" watchObservedRunningTime="2026-07-02 14:38:35.34430111 +0000 UTC m=+5026.572212284"
Jul 02 14:38:37.168353 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:37.168253    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-676677444b-x285c_21b94548-a1f6-42a4-be04-9de433ac4f8f/manager/0.log"
Jul 02 14:38:38.069484 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:38.069449    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-66f98455b5-tvvfz_31249163-a16c-4153-961c-6b50def8fd9b/manager/0.log"
Jul 02 14:38:38.091934 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:38.091902    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-jl6r2_4d680a83-8ceb-4eb0-b7fc-311c1e1ccc94/server/0.log"
Jul 02 14:38:39.045158 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:39.045111    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-qw5vv_2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0/s3-init/0.log"
Jul 02 14:38:39.061989 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:39.061961    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-qw5vv_2cfaa043-cd0e-416e-a6bb-84b0c6aa4ad0/download-hf-model/0.log"
Jul 02 14:38:45.381980 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:45.381937    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-2hwvd_d326b4cb-cbbe-41fc-8de2-e1876c60c9df/kube-multus/0.log"
Jul 02 14:38:45.613214 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:45.613176    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fjbsm_249b4a4b-a5eb-4bc1-8311-1c330010730d/kube-multus-additional-cni-plugins/0.log"
Jul 02 14:38:45.637145 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:45.637056    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fjbsm_249b4a4b-a5eb-4bc1-8311-1c330010730d/egress-router-binary-copy/0.log"
Jul 02 14:38:45.663463 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:45.663434    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fjbsm_249b4a4b-a5eb-4bc1-8311-1c330010730d/cni-plugins/0.log"
Jul 02 14:38:45.688875 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:45.688841    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fjbsm_249b4a4b-a5eb-4bc1-8311-1c330010730d/bond-cni-plugin/0.log"
Jul 02 14:38:45.712426 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:45.712349    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fjbsm_249b4a4b-a5eb-4bc1-8311-1c330010730d/routeoverride-cni/0.log"
Jul 02 14:38:45.734277 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:45.734243    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fjbsm_249b4a4b-a5eb-4bc1-8311-1c330010730d/whereabouts-cni-bincopy/0.log"
Jul 02 14:38:45.757286 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:45.757254    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fjbsm_249b4a4b-a5eb-4bc1-8311-1c330010730d/whereabouts-cni/0.log"
Jul 02 14:38:46.065206 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:46.065132    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xzl6q_0cae18b4-8f5b-4ff7-a73f-fc668af0a663/network-metrics-daemon/0.log"
Jul 02 14:38:46.083837 ip-10-0-135-143 kubenswrapper[2583]: I0702 14:38:46.083812    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xzl6q_0cae18b4-8f5b-4ff7-a73f-fc668af0a663/kube-rbac-proxy/0.log"