Sep 17 18:27:18.461471 ip-10-0-136-132 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 18:27:18.461485 ip-10-0-136-132 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 18:27:18.461494 ip-10-0-136-132 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 18:27:18.461804 ip-10-0-136-132 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 18:27:27.914306 ip-10-0-136-132 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 18:27:27.914322 ip-10-0-136-132 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 88c1d1b9bb984523b6478addc09e8a00 --
Sep 17 18:29:21.533920 ip-10-0-136-132 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 18:29:25.323594 ip-10-0-136-132 kubenswrapper[2588]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:29:25.323594 ip-10-0-136-132 kubenswrapper[2588]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 18:29:25.323594 ip-10-0-136-132 kubenswrapper[2588]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:29:25.323594 ip-10-0-136-132 kubenswrapper[2588]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 18:29:25.324747 ip-10-0-136-132 kubenswrapper[2588]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:29:25.324747 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.323706    2588 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 18:29:25.327981 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.327957    2588 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:25.327981 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.327976    2588 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:25.327981 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.327984    2588 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:25.327981 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.327988    2588 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:25.328256 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.327994    2588 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:25.328256 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328000    2588 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:25.328256 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328004    2588 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:25.328256 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328008    2588 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:25.328256 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328013    2588 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:25.328256 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328017    2588 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:25.328256 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328026    2588 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:25.328256 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328030    2588 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:25.328256 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328035    2588 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:25.328256 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328040    2588 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:25.328256 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328043    2588 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:25.328256 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328048    2588 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:25.328256 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328052    2588 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:25.328256 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328056    2588 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:25.328256 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328061    2588 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:25.328256 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328065    2588 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:25.328256 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328069    2588 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:25.328256 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328073    2588 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:25.328256 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328077    2588 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:25.328256 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328081    2588 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:25.329106 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328086    2588 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:25.329106 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328090    2588 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:25.329106 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328095    2588 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:25.329106 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328099    2588 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:25.329106 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328103    2588 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:25.329106 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328108    2588 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:25.329106 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328112    2588 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:25.329106 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328117    2588 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:25.329106 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328121    2588 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:25.329106 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328126    2588 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:25.329106 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328130    2588 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:25.329106 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328134    2588 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:25.329106 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328139    2588 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:25.329106 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328146    2588 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:25.329106 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328150    2588 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:25.329106 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328155    2588 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:25.329106 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328161    2588 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:25.329106 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328165    2588 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:25.329106 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328169    2588 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:25.329814 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328175    2588 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:25.329814 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328180    2588 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:25.329814 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328185    2588 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:25.329814 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328189    2588 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:25.329814 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328194    2588 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:25.329814 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328198    2588 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:25.329814 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328205    2588 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:25.329814 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328210    2588 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:25.329814 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328215    2588 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:25.329814 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328219    2588 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:25.329814 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328223    2588 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:25.329814 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328231    2588 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:25.329814 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328237    2588 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:25.329814 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328242    2588 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:25.329814 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328246    2588 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:25.329814 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328251    2588 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:25.329814 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328255    2588 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:25.329814 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328260    2588 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:25.329814 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328267    2588 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:25.330314 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328275    2588 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:25.330314 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328283    2588 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:25.330314 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328288    2588 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:25.330314 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328294    2588 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:25.330314 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328299    2588 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:25.330314 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328304    2588 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:25.330314 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328309    2588 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:25.330314 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328316    2588 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:25.330314 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328321    2588 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:25.330314 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328326    2588 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:25.330314 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328332    2588 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:25.330314 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328337    2588 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:25.330314 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328341    2588 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:25.330314 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328346    2588 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:25.330314 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328351    2588 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:25.330314 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328356    2588 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:25.330314 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328361    2588 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:25.330314 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328365    2588 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:25.330314 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328369    2588 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:25.330314 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328374    2588 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:25.330894 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328379    2588 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:25.330894 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328383    2588 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:25.330894 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328388    2588 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:25.330894 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328392    2588 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:25.330894 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328398    2588 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:25.330894 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328402    2588 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:25.330894 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328407    2588 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:25.330894 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328413    2588 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:25.330894 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328419    2588 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:25.330894 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328424    2588 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:25.330894 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328429    2588 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:25.330894 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328433    2588 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:25.330894 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328439    2588 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:25.330894 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328443    2588 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:25.330894 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328448    2588 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:25.330894 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328452    2588 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:25.330894 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.328457    2588 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:25.330894 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330838    2588 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:25.330894 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330851    2588 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:25.330894 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330855    2588 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:25.331469 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330859    2588 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:25.331469 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330862    2588 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:25.331469 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330865    2588 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:25.331469 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330868    2588 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:25.331469 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330871    2588 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:25.331469 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330875    2588 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:25.331469 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330878    2588 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:25.331469 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330881    2588 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:25.331469 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330884    2588 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:25.331469 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330887    2588 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:25.331469 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330890    2588 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:25.331469 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330893    2588 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:25.331469 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330896    2588 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:25.331469 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330899    2588 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:25.331469 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330902    2588 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:25.331469 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330905    2588 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:25.331469 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330909    2588 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:25.331469 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330912    2588 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:25.331469 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330914    2588 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:25.331469 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330917    2588 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:25.332022 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330920    2588 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:25.332022 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330922    2588 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:25.332022 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330925    2588 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:25.332022 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330928    2588 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:25.332022 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330931    2588 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:25.332022 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330937    2588 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:25.332022 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330942    2588 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:25.332022 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330945    2588 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:25.332022 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330948    2588 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:25.332022 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330951    2588 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:25.332022 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330954    2588 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:25.332022 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330957    2588 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:25.332022 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330960    2588 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:25.332022 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330963    2588 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:25.332022 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330966    2588 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:25.332022 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330969    2588 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:25.332022 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330972    2588 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:25.332022 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330977    2588 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:25.332500 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330980    2588 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:25.332500 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330984    2588 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:25.332500 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330987    2588 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:25.332500 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330990    2588 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:25.332500 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330992    2588 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:25.332500 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330996    2588 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:25.332500 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.330998    2588 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:25.332500 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331001    2588 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:25.332500 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331004    2588 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:25.332500 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331007    2588 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:25.332500 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331009    2588 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:25.332500 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331012    2588 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:25.332500 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331015    2588 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:25.332500 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331018    2588 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:25.332500 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331020    2588 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:25.332500 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331023    2588 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:25.332500 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331026    2588 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:25.332500 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331029    2588 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:25.332500 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331032    2588 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:25.333014 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331035    2588 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:25.333014 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331039    2588 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:25.333014 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331041    2588 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:25.333014 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331045    2588 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:25.333014 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331047    2588 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:25.333014 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331050    2588 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:25.333014 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331053    2588 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:25.333014 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331056    2588 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:25.333014 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331059    2588 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:25.333014 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331062    2588 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:25.333014 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331065    2588 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:25.333014 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331068    2588 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:25.333014 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331071    2588 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:25.333014 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331074    2588 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:25.333014 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331077    2588 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:25.333014 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331080    2588 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:25.333014 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331083    2588 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:25.333014 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331085    2588 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:25.333014 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331088    2588 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:25.333014 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331091    2588 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:25.333014 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331107    2588 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:25.333566 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331110    2588 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:25.333566 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331114    2588 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:25.333566 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331118    2588 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:25.333566 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331121    2588 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:25.333566 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331124    2588 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:25.333566 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331127    2588 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:25.333566 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331130    2588 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:25.333566 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331133    2588 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:25.333566 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331136    2588 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:25.333566 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331140    2588 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:25.333566 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331143    2588 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:25.333566 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331146    2588 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:25.333566 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331150    2588 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:25.333566 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331154    2588 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:25.333566 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331157    2588 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:25.333566 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331160    2588 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:25.333566 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331163    2588 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:25.333566 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331168    2588 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:25.333566 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331571    2588 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:25.333566 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331577    2588 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:25.334111 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331580    2588 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:25.334111 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331583    2588 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:25.334111 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331586    2588 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:25.334111 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331589    2588 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:25.334111 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331592    2588 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:25.334111 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331595    2588 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:25.334111 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331598    2588 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:25.334111 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331601    2588 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:25.334111 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331620    2588 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:25.334111 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331625    2588 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:25.334111 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331629    2588 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:25.334111 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331632    2588 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:25.334111 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331635    2588 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:25.334111 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331638    2588 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:25.334111 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331642    2588 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:25.334111 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331645    2588 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:25.334111 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331648    2588 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:25.334111 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331651    2588 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:25.334111 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331654    2588 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:25.334111 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331656    2588 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:25.334650 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331659    2588 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:25.334650 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331662    2588 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:25.334650 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331666    2588 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:25.334650 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331670    2588 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:25.334650 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331673    2588 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:25.334650 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331675    2588 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:25.334650 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331678    2588 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:25.334650 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331681    2588 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:25.334650 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331684    2588 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:25.334650 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331687    2588 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:25.334650 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331690    2588 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:25.334650 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331693    2588 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:25.334650 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331696    2588 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:25.334650 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331698    2588 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:25.334650 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331701    2588 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:25.334650 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331704    2588 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:25.334650 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331707    2588 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:25.334650 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331709    2588 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:25.334650 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331712    2588 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:25.334650 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331715    2588 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:25.335173 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331718    2588 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:25.335173 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331721    2588 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:25.335173 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331723    2588 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:25.335173 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331726    2588 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:25.335173 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331729    2588 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:25.335173 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331732    2588 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:25.335173 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331734    2588 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:25.335173 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331738    2588 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:25.335173 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331741    2588 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:25.335173 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331744    2588 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:25.335173 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331748    2588 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:25.335173 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331751    2588 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:25.335173 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331754    2588 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:25.335173 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331757    2588 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:25.335173 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331761    2588 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:25.335173 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331765    2588 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:25.335173 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331768    2588 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:25.335173 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331770    2588 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:25.335173 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331773    2588 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:25.335692 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331776    2588 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:25.335692 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331779    2588 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:25.335692 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331782    2588 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:25.335692 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331784    2588 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:25.335692 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331787    2588 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:25.335692 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331790    2588 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:25.335692 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331793    2588 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:25.335692 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331796    2588 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:25.335692 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331799    2588 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:25.335692 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331801    2588 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:25.335692 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331804    2588 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:25.335692 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331807    2588 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:25.335692 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331810    2588 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:25.335692 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331812    2588 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:25.335692 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331815    2588 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:25.335692 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331817    2588 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:25.335692 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331820    2588 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:25.335692 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331823    2588 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:25.335692 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331825    2588 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:25.335692 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331828    2588 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:25.336205 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331831    2588 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:25.336205 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331835    2588 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:25.336205 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331838    2588 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:25.336205 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331841    2588 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:25.336205 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331845    2588 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:25.336205 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331849    2588 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:25.336205 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331852    2588 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:25.336205 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331856    2588 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:25.336205 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331860    2588 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:25.336205 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331864    2588 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:25.336205 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331867    2588 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:25.336205 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331870    2588 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:25.336205 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331873    2588 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:25.336205 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331876    2588 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:25.336205 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331880    2588 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:25.336205 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331884    2588 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:25.336205 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331887    2588 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:25.336205 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.331889    2588 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:25.336205 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.331977    2588 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 18:29:25.336751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332027    2588 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 18:29:25.336751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332036    2588 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 18:29:25.336751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332041    2588 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 18:29:25.336751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332046    2588 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 18:29:25.336751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332049    2588 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 18:29:25.336751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332054    2588 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 18:29:25.336751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332058    2588 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 18:29:25.336751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332062    2588 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 18:29:25.336751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332065    2588 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 18:29:25.336751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332069    2588 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 18:29:25.336751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332073    2588 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 18:29:25.336751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332076    2588 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 18:29:25.336751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332080    2588 flags.go:64] FLAG: --cgroup-root=""
Sep 17 18:29:25.336751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332083    2588 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 18:29:25.336751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332086    2588 flags.go:64] FLAG: --client-ca-file=""
Sep 17 18:29:25.336751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332090    2588 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 18:29:25.336751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332094    2588 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 18:29:25.336751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332099    2588 flags.go:64] FLAG: --cluster-domain=""
Sep 17 18:29:25.336751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332102    2588 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 18:29:25.336751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332105    2588 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 18:29:25.336751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332113    2588 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 18:29:25.336751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332117    2588 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 18:29:25.336751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332121    2588 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 18:29:25.336751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332124    2588 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 18:29:25.337381 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332128    2588 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 18:29:25.337381 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332131    2588 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 18:29:25.337381 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332135    2588 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 18:29:25.337381 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332139    2588 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 18:29:25.337381 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332142    2588 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 18:29:25.337381 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332146    2588 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 18:29:25.337381 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332149    2588 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 18:29:25.337381 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332153    2588 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 18:29:25.337381 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332156    2588 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 18:29:25.337381 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332160    2588 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 18:29:25.337381 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332163    2588 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 18:29:25.337381 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332167    2588 flags.go:64] FLAG: --enable-server="true"
Sep 17 18:29:25.337381 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332170    2588 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 18:29:25.337381 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332175    2588 flags.go:64] FLAG: --event-burst="100"
Sep 17 18:29:25.337381 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332178    2588 flags.go:64] FLAG: --event-qps="50"
Sep 17 18:29:25.337381 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332181    2588 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 18:29:25.337381 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332184    2588 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 18:29:25.337381 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332188    2588 flags.go:64] FLAG: --eviction-hard=""
Sep 17 18:29:25.337381 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332192    2588 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 18:29:25.337381 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332195    2588 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 18:29:25.337381 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332198    2588 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 18:29:25.337381 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332201    2588 flags.go:64] FLAG: --eviction-soft=""
Sep 17 18:29:25.337381 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332205    2588 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 18:29:25.337381 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332208    2588 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 18:29:25.337381 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332211    2588 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 18:29:25.338064 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332215    2588 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 18:29:25.338064 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332218    2588 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 18:29:25.338064 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332222    2588 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 18:29:25.338064 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332227    2588 flags.go:64] FLAG: --feature-gates=""
Sep 17 18:29:25.338064 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332233    2588 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 18:29:25.338064 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332236    2588 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 18:29:25.338064 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332240    2588 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 18:29:25.338064 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332244    2588 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 18:29:25.338064 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332247    2588 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 18:29:25.338064 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332250    2588 flags.go:64] FLAG: --help="false"
Sep 17 18:29:25.338064 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332254    2588 flags.go:64] FLAG: --hostname-override="ip-10-0-136-132.ec2.internal"
Sep 17 18:29:25.338064 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332257    2588 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 18:29:25.338064 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332260    2588 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 18:29:25.338064 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332263    2588 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 18:29:25.338064 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332267    2588 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 18:29:25.338064 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332271    2588 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 18:29:25.338064 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332274    2588 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 18:29:25.338064 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332278    2588 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 18:29:25.338064 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332281    2588 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 18:29:25.338064 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332284    2588 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 18:29:25.338064 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332287    2588 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 18:29:25.338064 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332290    2588 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 18:29:25.338064 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332293    2588 flags.go:64] FLAG: --kube-reserved=""
Sep 17 18:29:25.338064 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332297    2588 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 18:29:25.338751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332300    2588 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 18:29:25.338751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332303    2588 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 18:29:25.338751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332306    2588 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 18:29:25.338751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332309    2588 flags.go:64] FLAG: --lock-file=""
Sep 17 18:29:25.338751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332312    2588 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 18:29:25.338751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332315    2588 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 18:29:25.338751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332319    2588 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 18:29:25.338751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332324    2588 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 18:29:25.338751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332328    2588 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 18:29:25.338751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332331    2588 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 18:29:25.338751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332334    2588 flags.go:64] FLAG: --logging-format="text"
Sep 17 18:29:25.338751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332339    2588 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 18:29:25.338751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332344    2588 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 18:29:25.338751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332347    2588 flags.go:64] FLAG: --manifest-url=""
Sep 17 18:29:25.338751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332351    2588 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 18:29:25.338751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332356    2588 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 18:29:25.338751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332359    2588 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 18:29:25.338751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332363    2588 flags.go:64] FLAG: --max-pods="110"
Sep 17 18:29:25.338751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332366    2588 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 18:29:25.338751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332369    2588 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 18:29:25.338751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332372    2588 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 18:29:25.338751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332376    2588 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 18:29:25.338751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332379    2588 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 18:29:25.338751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332382    2588 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 18:29:25.338751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332385    2588 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 18:29:25.339410 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332393    2588 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 18:29:25.339410 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332397    2588 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 18:29:25.339410 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332400    2588 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 18:29:25.339410 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332403    2588 flags.go:64] FLAG: --pod-cidr=""
Sep 17 18:29:25.339410 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332407    2588 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 18:29:25.339410 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332413    2588 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 18:29:25.339410 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332416    2588 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 18:29:25.339410 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332419    2588 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 18:29:25.339410 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332422    2588 flags.go:64] FLAG: --port="10250"
Sep 17 18:29:25.339410 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332426    2588 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 18:29:25.339410 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332429    2588 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03840e697721ba3f8"
Sep 17 18:29:25.339410 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332432    2588 flags.go:64] FLAG: --qos-reserved=""
Sep 17 18:29:25.339410 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332435    2588 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 18:29:25.339410 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332439    2588 flags.go:64] FLAG: --register-node="true"
Sep 17 18:29:25.339410 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332442    2588 flags.go:64] FLAG: --register-with-taints=""
Sep 17 18:29:25.339410 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332446    2588 flags.go:64] FLAG: --registry-burst="10"
Sep 17 18:29:25.339410 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332449    2588 flags.go:64] FLAG: --registry-qps="5"
Sep 17 18:29:25.339410 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332453    2588 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 18:29:25.339410 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332456    2588 flags.go:64] FLAG: --reserved-memory=""
Sep 17 18:29:25.339410 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332466    2588 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 18:29:25.339410 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332470    2588 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 18:29:25.339410 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332473    2588 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 18:29:25.339410 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332477    2588 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 18:29:25.339410 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332480    2588 flags.go:64] FLAG: --runonce="false"
Sep 17 18:29:25.339410 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332483    2588 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 18:29:25.340323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332486    2588 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 18:29:25.340323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332490    2588 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 18:29:25.340323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332493    2588 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 18:29:25.340323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332496    2588 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 18:29:25.340323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332500    2588 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 18:29:25.340323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332503    2588 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 18:29:25.340323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332507    2588 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 18:29:25.340323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332510    2588 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 18:29:25.340323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332514    2588 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 18:29:25.340323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332517    2588 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 18:29:25.340323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332520    2588 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 18:29:25.340323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332524    2588 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 18:29:25.340323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332527    2588 flags.go:64] FLAG: --system-cgroups=""
Sep 17 18:29:25.340323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332530    2588 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 18:29:25.340323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332536    2588 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 18:29:25.340323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332540    2588 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 18:29:25.340323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332543    2588 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 18:29:25.340323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332548    2588 flags.go:64] FLAG: --tls-min-version=""
Sep 17 18:29:25.340323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332551    2588 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 18:29:25.340323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332554    2588 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 18:29:25.340323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332557    2588 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 18:29:25.340323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332560    2588 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 18:29:25.340323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332563    2588 flags.go:64] FLAG: --v="2"
Sep 17 18:29:25.340323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332567    2588 flags.go:64] FLAG: --version="false"
Sep 17 18:29:25.340323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332573    2588 flags.go:64] FLAG: --vmodule=""
Sep 17 18:29:25.341501 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332577    2588 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 18:29:25.341501 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.332582    2588 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 18:29:25.341501 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332724    2588 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:25.341501 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332729    2588 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:25.341501 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332735    2588 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:25.341501 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332739    2588 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:25.341501 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332743    2588 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:25.341501 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332747    2588 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:25.341501 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332751    2588 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:25.341501 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332754    2588 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:25.341501 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332757    2588 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:25.341501 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332761    2588 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:25.341501 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332764    2588 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:25.341501 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332767    2588 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:25.341501 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332770    2588 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:25.341501 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332774    2588 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:25.341501 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332777    2588 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:25.341501 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332780    2588 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:25.341501 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332783    2588 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:25.341501 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332785    2588 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:25.342452 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332788    2588 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:25.342452 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332793    2588 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:25.342452 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332796    2588 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:25.342452 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332799    2588 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:25.342452 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332802    2588 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:25.342452 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332805    2588 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:25.342452 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332807    2588 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:25.342452 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332811    2588 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:25.342452 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332813    2588 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:25.342452 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332816    2588 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:25.342452 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332819    2588 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:25.342452 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332822    2588 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:25.342452 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332826    2588 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:25.342452 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332832    2588 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:25.342452 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332835    2588 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:25.342452 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332837    2588 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:25.342452 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332840    2588 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:25.342452 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332843    2588 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:25.342452 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332846    2588 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:25.343460 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332849    2588 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:25.343460 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332852    2588 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:25.343460 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332855    2588 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:25.343460 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332858    2588 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:25.343460 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332861    2588 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:25.343460 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332863    2588 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:25.343460 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332866    2588 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:25.343460 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332869    2588 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:25.343460 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332872    2588 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:25.343460 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332875    2588 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:25.343460 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332877    2588 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:25.343460 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332880    2588 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:25.343460 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332883    2588 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:25.343460 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332886    2588 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:25.343460 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332889    2588 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:25.343460 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332891    2588 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:25.343460 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332894    2588 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:25.343460 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332897    2588 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:25.343460 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332900    2588 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:25.344307 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332902    2588 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:25.344307 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332905    2588 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:25.344307 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332908    2588 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:25.344307 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332911    2588 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:25.344307 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332914    2588 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:25.344307 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332917    2588 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:25.344307 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332920    2588 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:25.344307 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332926    2588 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:25.344307 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332929    2588 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:25.344307 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332932    2588 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:25.344307 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332935    2588 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:25.344307 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332939    2588 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:25.344307 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332943    2588 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:25.344307 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332946    2588 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:25.344307 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332949    2588 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:25.344307 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332952    2588 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:25.344307 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332955    2588 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:25.344307 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332958    2588 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:25.344307 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332960    2588 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:25.344307 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332963    2588 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:25.345222 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332966    2588 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:25.345222 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332969    2588 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:25.345222 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332972    2588 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:25.345222 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332975    2588 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:25.345222 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332978    2588 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:25.345222 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332981    2588 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:25.345222 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332983    2588 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:25.345222 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332986    2588 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:25.345222 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332989    2588 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:25.345222 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332992    2588 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:25.345222 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332995    2588 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:25.345222 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.332997    2588 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:25.345222 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.333000    2588 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:25.345222 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.333003    2588 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:25.345222 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.333006    2588 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:25.345222 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.333009    2588 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:25.345222 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.333011    2588 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:25.345222 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.333014    2588 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:25.345222 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.333017    2588 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:25.345222 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.333023    2588 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:25.346069 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.333028    2588 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:25.346069 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.333031    2588 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:25.346069 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.333034    2588 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:25.346069 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.333785    2588 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:29:25.346069 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.342197    2588 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 18:29:25.346069 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.342217    2588 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 18:29:25.346069 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.342247    2588 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 18:29:25.346069 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.342257    2588 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 18:29:25.346069 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342348    2588 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:25.346069 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342356    2588 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:25.346069 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342361    2588 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:25.346069 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342369    2588 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:25.346069 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342375    2588 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:25.346069 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342381    2588 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:25.346459 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342386    2588 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:25.346459 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342391    2588 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:25.346459 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342396    2588 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:25.346459 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342401    2588 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:25.346459 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342406    2588 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:25.346459 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342410    2588 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:25.346459 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342415    2588 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:25.346459 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342419    2588 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:25.346459 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342424    2588 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:25.346459 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342429    2588 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:25.346459 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342434    2588 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:25.346459 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342439    2588 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:25.346459 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342443    2588 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:25.346459 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342447    2588 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:25.346459 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342452    2588 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:25.346459 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342457    2588 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:25.346459 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342461    2588 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:25.346459 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342466    2588 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:25.346459 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342471    2588 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:25.346459 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342475    2588 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:25.347191 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342480    2588 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:25.347191 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342485    2588 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:25.347191 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342490    2588 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:25.347191 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342495    2588 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:25.347191 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342500    2588 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:25.347191 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342505    2588 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:25.347191 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342510    2588 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:25.347191 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342515    2588 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:25.347191 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342520    2588 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:25.347191 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342524    2588 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:25.347191 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342529    2588 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:25.347191 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342533    2588 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:25.347191 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342538    2588 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:25.347191 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342543    2588 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:25.347191 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342548    2588 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:25.347191 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342553    2588 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:25.347191 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342557    2588 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:25.347191 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342562    2588 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:25.347191 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342567    2588 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:25.347191 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342572    2588 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:25.347191 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342576    2588 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:25.347832 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342581    2588 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:25.347832 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342585    2588 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:25.347832 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342590    2588 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:25.347832 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342594    2588 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:25.347832 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342616    2588 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:25.347832 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342621    2588 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:25.347832 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342626    2588 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:25.347832 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342631    2588 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:25.347832 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342638    2588 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:25.347832 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342644    2588 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:25.347832 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342651    2588 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:25.347832 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342658    2588 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:25.347832 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342664    2588 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:25.347832 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342669    2588 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:25.347832 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342674    2588 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:25.347832 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342679    2588 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:25.347832 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342684    2588 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:25.347832 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342689    2588 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:25.347832 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342694    2588 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:25.348337 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342698    2588 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:25.348337 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342703    2588 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:25.348337 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342707    2588 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:25.348337 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342711    2588 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:25.348337 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342716    2588 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:25.348337 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342720    2588 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:25.348337 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342725    2588 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:25.348337 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342729    2588 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:25.348337 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342733    2588 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:25.348337 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342739    2588 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:25.348337 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342744    2588 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:25.348337 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342748    2588 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:25.348337 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342753    2588 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:25.348337 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342758    2588 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:25.348337 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342762    2588 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:25.348337 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342767    2588 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:25.348337 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342771    2588 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:25.348337 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342776    2588 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:25.348337 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342780    2588 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:25.348946 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342785    2588 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:25.348946 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342790    2588 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:25.348946 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342795    2588 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:25.348946 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342809    2588 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:25.348946 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342813    2588 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:25.348946 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342819    2588 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:25.348946 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342824    2588 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:25.348946 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342829    2588 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:25.348946 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342834    2588 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:25.348946 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342840    2588 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:25.348946 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342845    2588 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:25.348946 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342850    2588 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:25.348946 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342855    2588 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:25.348946 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.342859    2588 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:25.348946 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.342868    2588 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:29:25.349379 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343073    2588 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:25.349379 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343081    2588 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:25.349379 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343086    2588 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:25.349379 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343091    2588 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:25.349379 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343098    2588 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:25.349379 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343105    2588 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:25.349379 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343110    2588 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:25.349379 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343115    2588 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:25.349379 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343120    2588 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:25.349379 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343124    2588 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:25.349379 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343129    2588 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:25.349379 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343133    2588 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:25.349379 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343137    2588 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:25.349379 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343142    2588 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:25.349379 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343147    2588 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:25.349379 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343151    2588 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:25.349379 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343156    2588 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:25.349379 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343160    2588 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:25.349379 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343165    2588 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:25.349997 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343170    2588 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:25.349997 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343174    2588 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:25.349997 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343179    2588 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:25.349997 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343187    2588 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:25.349997 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343196    2588 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:25.349997 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343201    2588 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:25.349997 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343207    2588 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:25.349997 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343213    2588 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:25.349997 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343217    2588 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:25.349997 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343222    2588 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:25.349997 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343226    2588 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:25.349997 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343231    2588 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:25.349997 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343235    2588 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:25.349997 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343240    2588 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:25.349997 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343244    2588 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:25.349997 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343249    2588 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:25.349997 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343253    2588 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:25.349997 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343258    2588 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:25.349997 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343262    2588 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:25.350902 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343267    2588 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:25.350902 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343271    2588 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:25.350902 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343275    2588 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:25.350902 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343280    2588 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:25.350902 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343287    2588 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:25.350902 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343292    2588 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:25.350902 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343298    2588 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:25.350902 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343303    2588 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:25.350902 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343309    2588 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:25.350902 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343314    2588 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:25.350902 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343319    2588 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:25.350902 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343324    2588 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:25.350902 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343329    2588 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:25.350902 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343333    2588 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:25.350902 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343338    2588 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:25.350902 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343343    2588 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:25.350902 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343347    2588 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:25.350902 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343352    2588 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:25.350902 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343357    2588 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:25.350902 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343361    2588 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:25.351841 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343366    2588 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:25.351841 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343371    2588 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:25.351841 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343375    2588 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:25.351841 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343380    2588 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:25.351841 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343384    2588 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:25.351841 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343389    2588 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:25.351841 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343393    2588 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:25.351841 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343398    2588 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:25.351841 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343402    2588 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:25.351841 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343407    2588 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:25.351841 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343411    2588 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:25.351841 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343415    2588 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:25.351841 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343420    2588 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:25.351841 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343425    2588 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:25.351841 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343429    2588 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:25.351841 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343434    2588 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:25.351841 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343438    2588 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:25.351841 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343443    2588 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:25.351841 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343447    2588 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:25.351841 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343451    2588 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:25.352816 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343455    2588 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:25.352816 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343460    2588 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:25.352816 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343464    2588 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:25.352816 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343468    2588 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:25.352816 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343472    2588 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:25.352816 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343477    2588 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:25.352816 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343481    2588 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:25.352816 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343485    2588 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:25.352816 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343490    2588 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:25.352816 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343494    2588 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:25.352816 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343498    2588 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:25.352816 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343504    2588 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:25.352816 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343509    2588 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:25.352816 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343513    2588 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:25.352816 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343519    2588 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:25.352816 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343523    2588 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:25.352816 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343528    2588 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:25.352816 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343533    2588 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:25.352816 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343537    2588 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:25.352816 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343541    2588 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:25.353553 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:25.343546    2588 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:25.353553 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.343554    2588 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:29:25.353553 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.343727    2588 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 18:29:25.353553 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.346626    2588 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 18:29:25.353553 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.347522    2588 server.go:1019] "Starting client certificate rotation"
Sep 17 18:29:25.353553 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.347634    2588 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 18:29:25.353553 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.347677    2588 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 18:29:25.353553 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.350467    2588 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 18:29:25.353553 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.352467    2588 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 18:29:25.364217 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.364195    2588 log.go:25] "Validated CRI v1 runtime API"
Sep 17 18:29:25.365958 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.365937    2588 log.go:25] "Validated CRI v1 image API"
Sep 17 18:29:25.366838 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.366817    2588 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 18:29:25.369313 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.369290    2588 fs.go:135] Filesystem UUIDs: map[32a6cd32-122c-44dd-a50a-9fcbfbbbb11b:/dev/nvme0n1p4 425153f8-d331-4be0-9f52-d58e54a6ec9e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 17 18:29:25.369394 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.369310    2588 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 18:29:25.374669 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.374486    2588 manager.go:216] Machine: {Timestamp:2026-09-17 18:29:25.37308059 +0000 UTC m=+2.224085983 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101535 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23869d04c61d818bdac6ab3decdf19 SystemUUID:ec23869d-04c6-1d81-8bda-c6ab3decdf19 BootID:88c1d1b9-bb98-4523-b647-8addc09e8a00 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:eb:26:2f:89 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:eb:26:2f:89 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:ed:38:4b:f0:59 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 18:29:25.374669 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.374663    2588 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 18:29:25.374782 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.374761    2588 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 18:29:25.376336 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.376302    2588 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 18:29:25.376480 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.376338    2588 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-132.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 18:29:25.377651 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.377639    2588 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 18:29:25.377689 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.377655    2588 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 18:29:25.377689 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.377664    2588 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 18:29:25.377689 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.377675    2588 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 18:29:25.377779 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.377756    2588 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 18:29:25.380624 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.380597    2588 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 18:29:25.380744 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.380734    2588 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 18:29:25.380785 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.380771    2588 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 18:29:25.380785 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.380783    2588 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 18:29:25.380860 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.380795    2588 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 18:29:25.380860 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.380805    2588 kubelet.go:398] "Adding apiserver pod source"
Sep 17 18:29:25.380860 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.380821    2588 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 18:29:25.381252 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.381240    2588 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 18:29:25.381287 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.381263    2588 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 18:29:25.383644 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.383626    2588 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 18:29:25.386342 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.386326    2588 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 18:29:25.386412 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.386359    2588 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 18:29:25.386643 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.386616    2588 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 18:29:25.387576 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.387562    2588 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 18:29:25.387647 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.387580    2588 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 18:29:25.387647 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.387587    2588 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 18:29:25.387647 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.387601    2588 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 18:29:25.387647 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.387614    2588 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 18:29:25.387647 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.387623    2588 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 18:29:25.387647 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.387640    2588 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 18:29:25.387647 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.387645    2588 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 18:29:25.387845 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.387652    2588 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 18:29:25.387845 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.387658    2588 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 18:29:25.387845 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.387672    2588 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 18:29:25.387845 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.387678    2588 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 18:29:25.387845 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.387704    2588 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 18:29:25.387845 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.387710    2588 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 18:29:25.392399 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.392378    2588 server.go:1268] "Started kubelet"
Sep 17 18:29:25.392515 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.392402    2588 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 18:29:25.392515 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.392417    2588 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-132.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 18:29:25.392664 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.392492    2588 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 18:29:25.392664 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.392564    2588 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 18:29:25.392825 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:25.392656    2588 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 18:29:25.392825 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:25.392756    2588 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-132.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 18:29:25.426771 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.426507    2588 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 18:29:25.427360 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.427328    2588 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sfg6p"
Sep 17 18:29:25.427795 ip-10-0-136-132 systemd[1]: Started Kubernetes Kubelet.
Sep 17 18:29:25.428519 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:25.427958    2588 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 18:29:25.429355 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.429263    2588 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 18:29:25.429595 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.429575    2588 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sfg6p"
Sep 17 18:29:25.431083 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:25.430214    2588 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-132.ec2.internal.18d62f2b67733cd2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-132.ec2.internal,UID:ip-10-0-136-132.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-132.ec2.internal,},FirstTimestamp:2026-09-17 18:29:25.392055506 +0000 UTC m=+2.243060958,LastTimestamp:2026-09-17 18:29:25.392055506 +0000 UTC m=+2.243060958,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-132.ec2.internal,}"
Sep 17 18:29:25.434037 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.434021    2588 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 18:29:25.434116 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.434044    2588 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 18:29:25.434179 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:25.434166    2588 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-132.ec2.internal\" not found"
Sep 17 18:29:25.434661 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.434643    2588 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 18:29:25.434661 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.434651    2588 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 18:29:25.434832 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.434670    2588 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 18:29:25.434832 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.434810    2588 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 18:29:25.434832 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.434819    2588 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 18:29:25.435230 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.435214    2588 factory.go:55] Registering systemd factory
Sep 17 18:29:25.435230 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.435233    2588 factory.go:223] Registration of the systemd container factory successfully
Sep 17 18:29:25.435544 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.435502    2588 factory.go:153] Registering CRI-O factory
Sep 17 18:29:25.435544 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.435526    2588 factory.go:223] Registration of the crio container factory successfully
Sep 17 18:29:25.435716 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.435628    2588 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 18:29:25.435716 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.435649    2588 factory.go:103] Registering Raw factory
Sep 17 18:29:25.435716 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.435667    2588 manager.go:1196] Started watching for new ooms in manager
Sep 17 18:29:25.436184 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.436167    2588 manager.go:349] Starting recovery of all containers
Sep 17 18:29:25.447564 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.447542    2588 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:25.448454 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.448435    2588 manager.go:354] Recovery completed
Sep 17 18:29:25.450480 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:25.450455    2588 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-132.ec2.internal\" not found" node="ip-10-0-136-132.ec2.internal"
Sep 17 18:29:25.454201 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.454186    2588 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:25.456745 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.456719    2588 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-132.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:25.456833 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.456759    2588 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-132.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:25.456833 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.456771    2588 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-132.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:25.457437 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.457421    2588 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 18:29:25.457522 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.457437    2588 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 18:29:25.457522 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.457463    2588 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 18:29:25.457647 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.457433    2588 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:25.458798 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.458781    2588 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-132.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:25.458889 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.458814    2588 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-132.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:25.458889 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.458885    2588 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-132.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:25.458987 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.458940    2588 policy_none.go:49] "None policy: Start"
Sep 17 18:29:25.458987 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.458959    2588 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 18:29:25.458987 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.458970    2588 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 18:29:25.460151 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.460134    2588 policy_none.go:47] "Start"
Sep 17 18:29:25.498194 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.498071    2588 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 18:29:25.500010 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.499982    2588 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 18:29:25.500110 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.500048    2588 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 18:29:25.500110 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.500084    2588 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 18:29:25.500215 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:25.500139    2588 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 17 18:29:25.502589 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.502566    2588 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:25.512212 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.512197    2588 manager.go:337] "Starting Device Plugin manager"
Sep 17 18:29:25.512319 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:25.512236    2588 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 18:29:25.512319 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.512247    2588 server.go:87] "Starting device plugin registration server"
Sep 17 18:29:25.512576 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.512563    2588 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 18:29:25.512657 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.512590    2588 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 18:29:25.512759 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.512743    2588 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 18:29:25.512840 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.512829    2588 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 18:29:25.512914 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.512843    2588 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 18:29:25.513550 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:25.513515    2588 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 18:29:25.513630 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:25.513597    2588 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-132.ec2.internal\" not found"
Sep 17 18:29:25.601319 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.601215    2588 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-132.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-132.ec2.internal"]
Sep 17 18:29:25.601319 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.601319    2588 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:25.602426 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.602408    2588 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-132.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:25.602500 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.602439    2588 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-132.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:25.602500 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.602450    2588 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-132.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:25.603976 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.603954    2588 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:25.604126 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.604111    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-132.ec2.internal"
Sep 17 18:29:25.604195 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.604142    2588 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:25.604785 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.604763    2588 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-132.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:25.604785 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.604772    2588 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-132.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:25.604936 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.604795    2588 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-132.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:25.604936 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.604799    2588 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-132.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:25.604936 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.604809    2588 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-132.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:25.604936 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.604810    2588 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-132.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:25.606191 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.606176    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-132.ec2.internal"
Sep 17 18:29:25.606238 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.606205    2588 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:25.606905 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.606888    2588 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-132.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:25.607001 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.606914    2588 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-132.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:25.607001 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.606925    2588 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-132.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:25.613653 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.613638    2588 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:25.614448 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.614429    2588 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-132.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:25.614521 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.614462    2588 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-132.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:25.614521 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.614474    2588 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-132.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:25.614521 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.614495    2588 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-132.ec2.internal"
Sep 17 18:29:25.623188 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.623171    2588 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-132.ec2.internal"
Sep 17 18:29:25.623247 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:25.623193    2588 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-132.ec2.internal\": node \"ip-10-0-136-132.ec2.internal\" not found"
Sep 17 18:29:25.637649 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:25.637626    2588 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-132.ec2.internal\" not found" node="ip-10-0-136-132.ec2.internal"
Sep 17 18:29:25.641630 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:25.641610    2588 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-132.ec2.internal\" not found"
Sep 17 18:29:25.642687 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:25.642669    2588 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-132.ec2.internal\" not found" node="ip-10-0-136-132.ec2.internal"
Sep 17 18:29:25.736491 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.736453    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8d590791640aecc72b49fc406401ef47-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-132.ec2.internal\" (UID: \"8d590791640aecc72b49fc406401ef47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-132.ec2.internal"
Sep 17 18:29:25.736491 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.736491    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d590791640aecc72b49fc406401ef47-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-132.ec2.internal\" (UID: \"8d590791640aecc72b49fc406401ef47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-132.ec2.internal"
Sep 17 18:29:25.736491 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.736510    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/189465409bf7b6f4cb62d3daabaf12f1-config\") pod \"kube-apiserver-proxy-ip-10-0-136-132.ec2.internal\" (UID: \"189465409bf7b6f4cb62d3daabaf12f1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-132.ec2.internal"
Sep 17 18:29:25.742662 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:25.742643    2588 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-132.ec2.internal\" not found"
Sep 17 18:29:25.836941 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.836900    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/189465409bf7b6f4cb62d3daabaf12f1-config\") pod \"kube-apiserver-proxy-ip-10-0-136-132.ec2.internal\" (UID: \"189465409bf7b6f4cb62d3daabaf12f1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-132.ec2.internal"
Sep 17 18:29:25.837053 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.837000    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/189465409bf7b6f4cb62d3daabaf12f1-config\") pod \"kube-apiserver-proxy-ip-10-0-136-132.ec2.internal\" (UID: \"189465409bf7b6f4cb62d3daabaf12f1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-132.ec2.internal"
Sep 17 18:29:25.837053 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.837038    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8d590791640aecc72b49fc406401ef47-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-132.ec2.internal\" (UID: \"8d590791640aecc72b49fc406401ef47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-132.ec2.internal"
Sep 17 18:29:25.837132 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.837059    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d590791640aecc72b49fc406401ef47-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-132.ec2.internal\" (UID: \"8d590791640aecc72b49fc406401ef47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-132.ec2.internal"
Sep 17 18:29:25.837132 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.837098    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d590791640aecc72b49fc406401ef47-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-132.ec2.internal\" (UID: \"8d590791640aecc72b49fc406401ef47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-132.ec2.internal"
Sep 17 18:29:25.837203 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.837128    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8d590791640aecc72b49fc406401ef47-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-132.ec2.internal\" (UID: \"8d590791640aecc72b49fc406401ef47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-132.ec2.internal"
Sep 17 18:29:25.843517 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:25.843481    2588 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-132.ec2.internal\" not found"
Sep 17 18:29:25.941307 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.941228    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-132.ec2.internal"
Sep 17 18:29:25.944575 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:25.944557    2588 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-132.ec2.internal\" not found"
Sep 17 18:29:25.946547 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:25.946343    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-132.ec2.internal"
Sep 17 18:29:26.045601 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:26.045556    2588 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-132.ec2.internal\" not found"
Sep 17 18:29:26.145834 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:26.145806    2588 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-132.ec2.internal\" not found"
Sep 17 18:29:26.246543 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:26.246431    2588 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-132.ec2.internal\" not found"
Sep 17 18:29:26.347008 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:26.346975    2588 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-132.ec2.internal\" not found"
Sep 17 18:29:26.347008 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:26.346982    2588 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 18:29:26.347506 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:26.347124    2588 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 18:29:26.347506 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:26.347165    2588 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 18:29:26.434613 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:26.434574    2588 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 18:24:25 +0000 UTC" deadline="2028-04-10 10:23:00.070674835 +0000 UTC"
Sep 17 18:29:26.434613 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:26.434605    2588 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13695h53m33.636072354s"
Sep 17 18:29:26.447961 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:26.447936    2588 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-132.ec2.internal\" not found"
Sep 17 18:29:26.548980 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:26.548948    2588 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-132.ec2.internal\" not found"
Sep 17 18:29:26.550354 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:26.550315    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod189465409bf7b6f4cb62d3daabaf12f1.slice/crio-3612416f46d9a05dd340b73c5a916b739a6dd636678fb19e00f52f7b4f8205cb WatchSource:0}: Error finding container 3612416f46d9a05dd340b73c5a916b739a6dd636678fb19e00f52f7b4f8205cb: Status 404 returned error can't find the container with id 3612416f46d9a05dd340b73c5a916b739a6dd636678fb19e00f52f7b4f8205cb
Sep 17 18:29:26.550731 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:26.550708    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d590791640aecc72b49fc406401ef47.slice/crio-6b6cc8392f0ad5f5fd1d3e3237af1eccdd01ef54be128a6dbd45e28a7f78ba02 WatchSource:0}: Error finding container 6b6cc8392f0ad5f5fd1d3e3237af1eccdd01ef54be128a6dbd45e28a7f78ba02: Status 404 returned error can't find the container with id 6b6cc8392f0ad5f5fd1d3e3237af1eccdd01ef54be128a6dbd45e28a7f78ba02
Sep 17 18:29:26.555624 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:26.555606    2588 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:29:26.641686 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:26.641653    2588 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:26.649775 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:26.649748    2588 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-132.ec2.internal\" not found"
Sep 17 18:29:26.750254 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:26.750223    2588 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-132.ec2.internal\" not found"
Sep 17 18:29:26.850722 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:26.850636    2588 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-132.ec2.internal\" not found"
Sep 17 18:29:26.951251 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:26.951211    2588 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-132.ec2.internal\" not found"
Sep 17 18:29:27.006620 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.006424    2588 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:27.035489 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.035457    2588 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-132.ec2.internal"
Sep 17 18:29:27.045233 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.045201    2588 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 18:29:27.046283 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.046260    2588 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-132.ec2.internal"
Sep 17 18:29:27.055483 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.055464    2588 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 18:29:27.382096 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.382067    2588 apiserver.go:52] "Watching apiserver"
Sep 17 18:29:27.389095 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.389038    2588 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 18:29:27.390067 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.390028    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-2t758","kube-system/kube-apiserver-proxy-ip-10-0-136-132.ec2.internal","openshift-cluster-node-tuning-operator/tuned-z97dh","openshift-dns/node-resolver-745b6","openshift-multus/multus-f96pg","openshift-network-diagnostics/network-check-target-cxndc","openshift-network-operator/iptables-alerter-qsksv","kube-system/global-pull-secret-syncer-fs688","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kh6l","openshift-image-registry/node-ca-5cb86","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-132.ec2.internal","openshift-multus/multus-additional-cni-plugins-kpswk","openshift-multus/network-metrics-daemon-bnq8f","openshift-ovn-kubernetes/ovnkube-node-2rqhq"]
Sep 17 18:29:27.392242 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.392219    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qsksv"
Sep 17 18:29:27.393612 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.393439    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bnq8f"
Sep 17 18:29:27.393759 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:27.393521    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bnq8f" podUID="1c01ccd8-9e9c-467f-a2af-50720f272038"
Sep 17 18:29:27.394806 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.394770    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 18:29:27.394806 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.394804    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.395119 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.394986    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:29:27.395119 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.395015    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hrq5n\""
Sep 17 18:29:27.395119 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.395020    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 18:29:27.396135 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.395948    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-745b6"
Sep 17 18:29:27.396930 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.396723    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:29:27.396930 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.396829    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-htfv9\""
Sep 17 18:29:27.397542 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.397126    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.397542 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.397235    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 18:29:27.397903 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.397882    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 18:29:27.398352 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.398219    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 18:29:27.398512 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.398492    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxndc"
Sep 17 18:29:27.398618 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:27.398577    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxndc" podUID="9530dc1d-51a9-4c20-aa0f-dbfbf912e02e"
Sep 17 18:29:27.398868 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.398851    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-whgvr\""
Sep 17 18:29:27.399383 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.399352    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 18:29:27.399474 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.399460    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 18:29:27.399785 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.399760    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 18:29:27.399785 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.399771    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kh6l"
Sep 17 18:29:27.399938 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.399917    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-td6p7\""
Sep 17 18:29:27.400051 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.399771    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 18:29:27.401009 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.400991    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2t758"
Sep 17 18:29:27.401827 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.401807    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 18:29:27.401925 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.401885    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tz82f\""
Sep 17 18:29:27.402183 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.402164    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 18:29:27.402544 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.402433    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.403072 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.403054    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 18:29:27.403152 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.403087    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 18:29:27.403210 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.403154    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hntf5\""
Sep 17 18:29:27.403210 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.403172    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 18:29:27.403943 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.403923    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5cb86"
Sep 17 18:29:27.404473 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.404455    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 18:29:27.404620 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.404463    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 18:29:27.405386 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.405368    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 18:29:27.405454 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.405386    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 18:29:27.405509 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.405464    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 18:29:27.405693 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.405666    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cz4k5\""
Sep 17 18:29:27.405767 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.405736    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:29:27.405830 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:27.405787    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs688" podUID="c554cd64-2695-47d3-999a-2d8bf8d7ef63"
Sep 17 18:29:27.406017 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.405999    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 18:29:27.406069 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.406025    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 18:29:27.406069 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.406053    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 18:29:27.406542 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.406502    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f6qmj\""
Sep 17 18:29:27.406663 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.406557    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 18:29:27.406974 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.406956    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kpswk"
Sep 17 18:29:27.409141 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.409122    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 18:29:27.409323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.409297    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 18:29:27.409323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.409311    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nvpw8\""
Sep 17 18:29:27.422984 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.422963    2588 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:27.434745 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.434724    2588 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 18:29:27.436126 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.436108    2588 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 18:29:27.442561 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.442543    2588 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:27.447687 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.447663    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/61513726-d6fb-4e35-a39f-818bcb1a9596-hosts-file\") pod \"node-resolver-745b6\" (UID: \"61513726-d6fb-4e35-a39f-818bcb1a9596\") " pod="openshift-dns/node-resolver-745b6"
Sep 17 18:29:27.447796 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.447701    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c69432b1-9a81-4293-bd1c-4df1df813ba4-env-overrides\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.447796 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.447728    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-multus-cni-dir\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.447796 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.447751    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-multus-conf-dir\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.447796 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.447776    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/947781e1-4423-435e-b44f-53ca352f8df7-host\") pod \"node-ca-5cb86\" (UID: \"947781e1-4423-435e-b44f-53ca352f8df7\") " pod="openshift-image-registry/node-ca-5cb86"
Sep 17 18:29:27.447942 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.447801    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a7f6c88f-5861-4089-a4a8-75bb85c16c98-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kpswk\" (UID: \"a7f6c88f-5861-4089-a4a8-75bb85c16c98\") " pod="openshift-multus/multus-additional-cni-plugins-kpswk"
Sep 17 18:29:27.447942 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.447825    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skqcl\" (UniqueName: \"kubernetes.io/projected/61513726-d6fb-4e35-a39f-818bcb1a9596-kube-api-access-skqcl\") pod \"node-resolver-745b6\" (UID: \"61513726-d6fb-4e35-a39f-818bcb1a9596\") " pod="openshift-dns/node-resolver-745b6"
Sep 17 18:29:27.447942 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.447848    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5skqc\" (UniqueName: \"kubernetes.io/projected/f6048ce0-9af4-4205-baa1-c28be5018d95-kube-api-access-5skqc\") pod \"aws-ebs-csi-driver-node-2kh6l\" (UID: \"f6048ce0-9af4-4205-baa1-c28be5018d95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kh6l"
Sep 17 18:29:27.447942 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.447870    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-host-run-netns\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.447942 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.447893    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-multus-socket-dir-parent\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.447942 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.447929    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-host-var-lib-cni-bin\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.448175 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.447963    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c554cd64-2695-47d3-999a-2d8bf8d7ef63-original-pull-secret\") pod \"global-pull-secret-syncer-fs688\" (UID: \"c554cd64-2695-47d3-999a-2d8bf8d7ef63\") " pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:29:27.448175 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.447990    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a7f6c88f-5861-4089-a4a8-75bb85c16c98-cnibin\") pod \"multus-additional-cni-plugins-kpswk\" (UID: \"a7f6c88f-5861-4089-a4a8-75bb85c16c98\") " pod="openshift-multus/multus-additional-cni-plugins-kpswk"
Sep 17 18:29:27.448175 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448018    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk9wx\" (UniqueName: \"kubernetes.io/projected/1c01ccd8-9e9c-467f-a2af-50720f272038-kube-api-access-fk9wx\") pod \"network-metrics-daemon-bnq8f\" (UID: \"1c01ccd8-9e9c-467f-a2af-50720f272038\") " pod="openshift-multus/network-metrics-daemon-bnq8f"
Sep 17 18:29:27.448175 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448034    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-run-ovn\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.448175 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448049    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w6nh\" (UniqueName: \"kubernetes.io/projected/a7f6c88f-5861-4089-a4a8-75bb85c16c98-kube-api-access-8w6nh\") pod \"multus-additional-cni-plugins-kpswk\" (UID: \"a7f6c88f-5861-4089-a4a8-75bb85c16c98\") " pod="openshift-multus/multus-additional-cni-plugins-kpswk"
Sep 17 18:29:27.448175 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448064    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7969c2b7-d790-4dbd-bb84-4886d90f38d7-host-slash\") pod \"iptables-alerter-qsksv\" (UID: \"7969c2b7-d790-4dbd-bb84-4886d90f38d7\") " pod="openshift-network-operator/iptables-alerter-qsksv"
Sep 17 18:29:27.448175 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448084    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c01ccd8-9e9c-467f-a2af-50720f272038-metrics-certs\") pod \"network-metrics-daemon-bnq8f\" (UID: \"1c01ccd8-9e9c-467f-a2af-50720f272038\") " pod="openshift-multus/network-metrics-daemon-bnq8f"
Sep 17 18:29:27.448175 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448100    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-sys\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.448484 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448277    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-lib-modules\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.448484 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448309    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f6048ce0-9af4-4205-baa1-c28be5018d95-etc-selinux\") pod \"aws-ebs-csi-driver-node-2kh6l\" (UID: \"f6048ce0-9af4-4205-baa1-c28be5018d95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kh6l"
Sep 17 18:29:27.448484 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448336    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c69432b1-9a81-4293-bd1c-4df1df813ba4-ovn-node-metrics-cert\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.448484 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448372    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8462659d-7d75-4d56-b4c1-75dfc4aea577-multus-daemon-config\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.448484 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448408    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-etc-kubernetes\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.448484 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448427    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f6048ce0-9af4-4205-baa1-c28be5018d95-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2kh6l\" (UID: \"f6048ce0-9af4-4205-baa1-c28be5018d95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kh6l"
Sep 17 18:29:27.448484 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448441    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8462659d-7d75-4d56-b4c1-75dfc4aea577-cni-binary-copy\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.448484 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448454    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-host-run-netns\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.448484 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448473    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/947781e1-4423-435e-b44f-53ca352f8df7-serviceca\") pod \"node-ca-5cb86\" (UID: \"947781e1-4423-435e-b44f-53ca352f8df7\") " pod="openshift-image-registry/node-ca-5cb86"
Sep 17 18:29:27.448484 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448488    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-etc-sysconfig\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.448966 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448506    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-etc-sysctl-d\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.448966 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448546    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-etc-tuned\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.448966 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448570    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f6048ce0-9af4-4205-baa1-c28be5018d95-registration-dir\") pod \"aws-ebs-csi-driver-node-2kh6l\" (UID: \"f6048ce0-9af4-4205-baa1-c28be5018d95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kh6l"
Sep 17 18:29:27.448966 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448597    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks65r\" (UniqueName: \"kubernetes.io/projected/c69432b1-9a81-4293-bd1c-4df1df813ba4-kube-api-access-ks65r\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.448966 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448618    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhp2j\" (UniqueName: \"kubernetes.io/projected/7969c2b7-d790-4dbd-bb84-4886d90f38d7-kube-api-access-dhp2j\") pod \"iptables-alerter-qsksv\" (UID: \"7969c2b7-d790-4dbd-bb84-4886d90f38d7\") " pod="openshift-network-operator/iptables-alerter-qsksv"
Sep 17 18:29:27.448966 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448658    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f6048ce0-9af4-4205-baa1-c28be5018d95-device-dir\") pod \"aws-ebs-csi-driver-node-2kh6l\" (UID: \"f6048ce0-9af4-4205-baa1-c28be5018d95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kh6l"
Sep 17 18:29:27.448966 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448697    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-host-slash\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.448966 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448720    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-node-log\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.448966 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448743    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-system-cni-dir\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.448966 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448767    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7f6c88f-5861-4089-a4a8-75bb85c16c98-system-cni-dir\") pod \"multus-additional-cni-plugins-kpswk\" (UID: \"a7f6c88f-5861-4089-a4a8-75bb85c16c98\") " pod="openshift-multus/multus-additional-cni-plugins-kpswk"
Sep 17 18:29:27.448966 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448791    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a7f6c88f-5861-4089-a4a8-75bb85c16c98-cni-binary-copy\") pod \"multus-additional-cni-plugins-kpswk\" (UID: \"a7f6c88f-5861-4089-a4a8-75bb85c16c98\") " pod="openshift-multus/multus-additional-cni-plugins-kpswk"
Sep 17 18:29:27.448966 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448818    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a7f6c88f-5861-4089-a4a8-75bb85c16c98-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kpswk\" (UID: \"a7f6c88f-5861-4089-a4a8-75bb85c16c98\") " pod="openshift-multus/multus-additional-cni-plugins-kpswk"
Sep 17 18:29:27.448966 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448842    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-etc-modprobe-d\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.448966 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448864    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-host\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.448966 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448887    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61513726-d6fb-4e35-a39f-818bcb1a9596-tmp-dir\") pod \"node-resolver-745b6\" (UID: \"61513726-d6fb-4e35-a39f-818bcb1a9596\") " pod="openshift-dns/node-resolver-745b6"
Sep 17 18:29:27.448966 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448911    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-host-run-k8s-cni-cncf-io\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.449710 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448938    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-etc-kubernetes\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.449710 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448962    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-cnibin\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.449710 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.448983    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-host-var-lib-cni-multus\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.449710 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449007    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/10d63418-937e-4dee-a71f-592cfdff55ca-konnectivity-ca\") pod \"konnectivity-agent-2t758\" (UID: \"10d63418-937e-4dee-a71f-592cfdff55ca\") " pod="kube-system/konnectivity-agent-2t758"
Sep 17 18:29:27.449710 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449029    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-etc-systemd\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.449710 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449050    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-tmp\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.449710 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449072    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-host-run-multus-certs\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.449710 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449095    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f6048ce0-9af4-4205-baa1-c28be5018d95-sys-fs\") pod \"aws-ebs-csi-driver-node-2kh6l\" (UID: \"f6048ce0-9af4-4205-baa1-c28be5018d95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kh6l"
Sep 17 18:29:27.449710 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449119    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-run-openvswitch\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.449710 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449146    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c69432b1-9a81-4293-bd1c-4df1df813ba4-ovnkube-script-lib\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.449710 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449167    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-hostroot\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.449710 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449192    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnj7l\" (UniqueName: \"kubernetes.io/projected/8462659d-7d75-4d56-b4c1-75dfc4aea577-kube-api-access-wnj7l\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.449710 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449217    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/10d63418-937e-4dee-a71f-592cfdff55ca-agent-certs\") pod \"konnectivity-agent-2t758\" (UID: \"10d63418-937e-4dee-a71f-592cfdff55ca\") " pod="kube-system/konnectivity-agent-2t758"
Sep 17 18:29:27.449710 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449241    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mbkj\" (UniqueName: \"kubernetes.io/projected/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-kube-api-access-4mbkj\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.449710 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449264    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-var-lib-openvswitch\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.449710 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449288    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-var-lib-kubelet\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.449710 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449310    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-host-cni-netd\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.450356 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449337    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.450356 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449363    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c69432b1-9a81-4293-bd1c-4df1df813ba4-ovnkube-config\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.450356 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449393    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c554cd64-2695-47d3-999a-2d8bf8d7ef63-kubelet-config\") pod \"global-pull-secret-syncer-fs688\" (UID: \"c554cd64-2695-47d3-999a-2d8bf8d7ef63\") " pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:29:27.450356 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449416    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-run\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.450356 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449438    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f6048ce0-9af4-4205-baa1-c28be5018d95-socket-dir\") pod \"aws-ebs-csi-driver-node-2kh6l\" (UID: \"f6048ce0-9af4-4205-baa1-c28be5018d95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kh6l"
Sep 17 18:29:27.450356 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449461    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-etc-openvswitch\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.450356 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449483    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-log-socket\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.450356 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449516    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-host-run-ovn-kubernetes\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.450356 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449564    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c554cd64-2695-47d3-999a-2d8bf8d7ef63-dbus\") pod \"global-pull-secret-syncer-fs688\" (UID: \"c554cd64-2695-47d3-999a-2d8bf8d7ef63\") " pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:29:27.450356 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449570    2588 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 18:29:27.450356 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449610    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a7f6c88f-5861-4089-a4a8-75bb85c16c98-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kpswk\" (UID: \"a7f6c88f-5861-4089-a4a8-75bb85c16c98\") " pod="openshift-multus/multus-additional-cni-plugins-kpswk"
Sep 17 18:29:27.450356 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449702    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7969c2b7-d790-4dbd-bb84-4886d90f38d7-iptables-alerter-script\") pod \"iptables-alerter-qsksv\" (UID: \"7969c2b7-d790-4dbd-bb84-4886d90f38d7\") " pod="openshift-network-operator/iptables-alerter-qsksv"
Sep 17 18:29:27.450356 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449745    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-host-kubelet\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.450356 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449770    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-host-cni-bin\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.450356 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449796    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zwcs\" (UniqueName: \"kubernetes.io/projected/947781e1-4423-435e-b44f-53ca352f8df7-kube-api-access-4zwcs\") pod \"node-ca-5cb86\" (UID: \"947781e1-4423-435e-b44f-53ca352f8df7\") " pod="openshift-image-registry/node-ca-5cb86"
Sep 17 18:29:27.450356 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449831    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcwlc\" (UniqueName: \"kubernetes.io/projected/9530dc1d-51a9-4c20-aa0f-dbfbf912e02e-kube-api-access-zcwlc\") pod \"network-check-target-cxndc\" (UID: \"9530dc1d-51a9-4c20-aa0f-dbfbf912e02e\") " pod="openshift-network-diagnostics/network-check-target-cxndc"
Sep 17 18:29:27.450921 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449863    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-systemd-units\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.450921 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449892    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-run-systemd\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.450921 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449921    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-host-var-lib-kubelet\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.450921 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.449970    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-etc-sysctl-conf\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.474291 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.474268    2588 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jvgx2"
Sep 17 18:29:27.480202 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.480179    2588 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jvgx2"
Sep 17 18:29:27.504695 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.504636    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-132.ec2.internal" event={"ID":"8d590791640aecc72b49fc406401ef47","Type":"ContainerStarted","Data":"6b6cc8392f0ad5f5fd1d3e3237af1eccdd01ef54be128a6dbd45e28a7f78ba02"}
Sep 17 18:29:27.505858 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.505827    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-132.ec2.internal" event={"ID":"189465409bf7b6f4cb62d3daabaf12f1","Type":"ContainerStarted","Data":"3612416f46d9a05dd340b73c5a916b739a6dd636678fb19e00f52f7b4f8205cb"}
Sep 17 18:29:27.550350 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.550316    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk9wx\" (UniqueName: \"kubernetes.io/projected/1c01ccd8-9e9c-467f-a2af-50720f272038-kube-api-access-fk9wx\") pod \"network-metrics-daemon-bnq8f\" (UID: \"1c01ccd8-9e9c-467f-a2af-50720f272038\") " pod="openshift-multus/network-metrics-daemon-bnq8f"
Sep 17 18:29:27.550350 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.550352    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-run-ovn\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.550582 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.550379    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w6nh\" (UniqueName: \"kubernetes.io/projected/a7f6c88f-5861-4089-a4a8-75bb85c16c98-kube-api-access-8w6nh\") pod \"multus-additional-cni-plugins-kpswk\" (UID: \"a7f6c88f-5861-4089-a4a8-75bb85c16c98\") " pod="openshift-multus/multus-additional-cni-plugins-kpswk"
Sep 17 18:29:27.550582 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.550402    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7969c2b7-d790-4dbd-bb84-4886d90f38d7-host-slash\") pod \"iptables-alerter-qsksv\" (UID: \"7969c2b7-d790-4dbd-bb84-4886d90f38d7\") " pod="openshift-network-operator/iptables-alerter-qsksv"
Sep 17 18:29:27.550582 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.550424    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c01ccd8-9e9c-467f-a2af-50720f272038-metrics-certs\") pod \"network-metrics-daemon-bnq8f\" (UID: \"1c01ccd8-9e9c-467f-a2af-50720f272038\") " pod="openshift-multus/network-metrics-daemon-bnq8f"
Sep 17 18:29:27.550582 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.550444    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-sys\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.550582 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.550490    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-lib-modules\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.550582 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:27.550523    2588 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:27.550582 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.550540    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7969c2b7-d790-4dbd-bb84-4886d90f38d7-host-slash\") pod \"iptables-alerter-qsksv\" (UID: \"7969c2b7-d790-4dbd-bb84-4886d90f38d7\") " pod="openshift-network-operator/iptables-alerter-qsksv"
Sep 17 18:29:27.550582 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.550545    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f6048ce0-9af4-4205-baa1-c28be5018d95-etc-selinux\") pod \"aws-ebs-csi-driver-node-2kh6l\" (UID: \"f6048ce0-9af4-4205-baa1-c28be5018d95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kh6l"
Sep 17 18:29:27.551078 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:27.550605    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c01ccd8-9e9c-467f-a2af-50720f272038-metrics-certs podName:1c01ccd8-9e9c-467f-a2af-50720f272038 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:28.050584344 +0000 UTC m=+4.901589735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c01ccd8-9e9c-467f-a2af-50720f272038-metrics-certs") pod "network-metrics-daemon-bnq8f" (UID: "1c01ccd8-9e9c-467f-a2af-50720f272038") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:27.551078 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.550623    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c69432b1-9a81-4293-bd1c-4df1df813ba4-ovn-node-metrics-cert\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.551078 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.550629    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-sys\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.551078 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.550656    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f6048ce0-9af4-4205-baa1-c28be5018d95-etc-selinux\") pod \"aws-ebs-csi-driver-node-2kh6l\" (UID: \"f6048ce0-9af4-4205-baa1-c28be5018d95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kh6l"
Sep 17 18:29:27.551078 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.550656    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8462659d-7d75-4d56-b4c1-75dfc4aea577-multus-daemon-config\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.551078 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.550640    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-run-ovn\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.551078 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.550730    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-etc-kubernetes\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.551078 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.550772    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f6048ce0-9af4-4205-baa1-c28be5018d95-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2kh6l\" (UID: \"f6048ce0-9af4-4205-baa1-c28be5018d95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kh6l"
Sep 17 18:29:27.551078 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.550778    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-lib-modules\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.551078 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.550817    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-etc-kubernetes\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.551603 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.551241    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8462659d-7d75-4d56-b4c1-75dfc4aea577-cni-binary-copy\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.551603 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.551258    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-host-run-netns\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.551603 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.551288    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/947781e1-4423-435e-b44f-53ca352f8df7-serviceca\") pod \"node-ca-5cb86\" (UID: \"947781e1-4423-435e-b44f-53ca352f8df7\") " pod="openshift-image-registry/node-ca-5cb86"
Sep 17 18:29:27.551603 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.551302    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-etc-sysconfig\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.551603 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.551316    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-etc-sysctl-d\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.551603 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.551337    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8462659d-7d75-4d56-b4c1-75dfc4aea577-multus-daemon-config\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.551603 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.551331    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-etc-tuned\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.551603 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.551419    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f6048ce0-9af4-4205-baa1-c28be5018d95-registration-dir\") pod \"aws-ebs-csi-driver-node-2kh6l\" (UID: \"f6048ce0-9af4-4205-baa1-c28be5018d95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kh6l"
Sep 17 18:29:27.551603 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.550929    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f6048ce0-9af4-4205-baa1-c28be5018d95-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2kh6l\" (UID: \"f6048ce0-9af4-4205-baa1-c28be5018d95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kh6l"
Sep 17 18:29:27.551603 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.551442    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-host-run-netns\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.551603 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.551478    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-etc-sysconfig\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.551603 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.551476    2588 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 18:29:27.551603 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.551578    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f6048ce0-9af4-4205-baa1-c28be5018d95-registration-dir\") pod \"aws-ebs-csi-driver-node-2kh6l\" (UID: \"f6048ce0-9af4-4205-baa1-c28be5018d95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kh6l"
Sep 17 18:29:27.551603 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.551602    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks65r\" (UniqueName: \"kubernetes.io/projected/c69432b1-9a81-4293-bd1c-4df1df813ba4-kube-api-access-ks65r\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.552248 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.551626    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhp2j\" (UniqueName: \"kubernetes.io/projected/7969c2b7-d790-4dbd-bb84-4886d90f38d7-kube-api-access-dhp2j\") pod \"iptables-alerter-qsksv\" (UID: \"7969c2b7-d790-4dbd-bb84-4886d90f38d7\") " pod="openshift-network-operator/iptables-alerter-qsksv"
Sep 17 18:29:27.552248 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.551729    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f6048ce0-9af4-4205-baa1-c28be5018d95-device-dir\") pod \"aws-ebs-csi-driver-node-2kh6l\" (UID: \"f6048ce0-9af4-4205-baa1-c28be5018d95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kh6l"
Sep 17 18:29:27.552248 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.551762    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-host-slash\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.552248 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.551786    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-node-log\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.552248 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.551809    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-system-cni-dir\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.552248 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.551829    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-etc-sysctl-d\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.552248 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.551833    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7f6c88f-5861-4089-a4a8-75bb85c16c98-system-cni-dir\") pod \"multus-additional-cni-plugins-kpswk\" (UID: \"a7f6c88f-5861-4089-a4a8-75bb85c16c98\") " pod="openshift-multus/multus-additional-cni-plugins-kpswk"
Sep 17 18:29:27.552248 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.551871    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8462659d-7d75-4d56-b4c1-75dfc4aea577-cni-binary-copy\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.552248 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.551875    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a7f6c88f-5861-4089-a4a8-75bb85c16c98-cni-binary-copy\") pod \"multus-additional-cni-plugins-kpswk\" (UID: \"a7f6c88f-5861-4089-a4a8-75bb85c16c98\") " pod="openshift-multus/multus-additional-cni-plugins-kpswk"
Sep 17 18:29:27.552248 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.551878    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/947781e1-4423-435e-b44f-53ca352f8df7-serviceca\") pod \"node-ca-5cb86\" (UID: \"947781e1-4423-435e-b44f-53ca352f8df7\") " pod="openshift-image-registry/node-ca-5cb86"
Sep 17 18:29:27.552248 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.551872    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-host-slash\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.552248 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.551900    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-node-log\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.552248 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.551906    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a7f6c88f-5861-4089-a4a8-75bb85c16c98-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kpswk\" (UID: \"a7f6c88f-5861-4089-a4a8-75bb85c16c98\") " pod="openshift-multus/multus-additional-cni-plugins-kpswk"
Sep 17 18:29:27.552248 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.551832    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f6048ce0-9af4-4205-baa1-c28be5018d95-device-dir\") pod \"aws-ebs-csi-driver-node-2kh6l\" (UID: \"f6048ce0-9af4-4205-baa1-c28be5018d95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kh6l"
Sep 17 18:29:27.552248 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.551909    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-system-cni-dir\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.552248 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.551949    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-etc-modprobe-d\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.552248 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.551954    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7f6c88f-5861-4089-a4a8-75bb85c16c98-system-cni-dir\") pod \"multus-additional-cni-plugins-kpswk\" (UID: \"a7f6c88f-5861-4089-a4a8-75bb85c16c98\") " pod="openshift-multus/multus-additional-cni-plugins-kpswk"
Sep 17 18:29:27.553166 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.551976    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-host\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.553166 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552002    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61513726-d6fb-4e35-a39f-818bcb1a9596-tmp-dir\") pod \"node-resolver-745b6\" (UID: \"61513726-d6fb-4e35-a39f-818bcb1a9596\") " pod="openshift-dns/node-resolver-745b6"
Sep 17 18:29:27.553166 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552027    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-host-run-k8s-cni-cncf-io\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.553166 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552060    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-etc-modprobe-d\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.553166 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552093    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-etc-kubernetes\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.553166 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552134    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-cnibin\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.553166 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552143    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-etc-kubernetes\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.553166 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552082    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-host-run-k8s-cni-cncf-io\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.553166 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552160    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-host-var-lib-cni-multus\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.553166 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552195    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-cnibin\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.553166 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552200    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/10d63418-937e-4dee-a71f-592cfdff55ca-konnectivity-ca\") pod \"konnectivity-agent-2t758\" (UID: \"10d63418-937e-4dee-a71f-592cfdff55ca\") " pod="kube-system/konnectivity-agent-2t758"
Sep 17 18:29:27.553166 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552225    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-etc-systemd\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.553166 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552247    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-tmp\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.553166 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552286    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-host-run-multus-certs\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.553166 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552310    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f6048ce0-9af4-4205-baa1-c28be5018d95-sys-fs\") pod \"aws-ebs-csi-driver-node-2kh6l\" (UID: \"f6048ce0-9af4-4205-baa1-c28be5018d95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kh6l"
Sep 17 18:29:27.553166 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552324    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61513726-d6fb-4e35-a39f-818bcb1a9596-tmp-dir\") pod \"node-resolver-745b6\" (UID: \"61513726-d6fb-4e35-a39f-818bcb1a9596\") " pod="openshift-dns/node-resolver-745b6"
Sep 17 18:29:27.553166 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552333    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-run-openvswitch\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.553166 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552374    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c69432b1-9a81-4293-bd1c-4df1df813ba4-ovnkube-script-lib\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.553951 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552387    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a7f6c88f-5861-4089-a4a8-75bb85c16c98-cni-binary-copy\") pod \"multus-additional-cni-plugins-kpswk\" (UID: \"a7f6c88f-5861-4089-a4a8-75bb85c16c98\") " pod="openshift-multus/multus-additional-cni-plugins-kpswk"
Sep 17 18:29:27.553951 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552397    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-hostroot\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.553951 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552226    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-host-var-lib-cni-multus\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.553951 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552437    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnj7l\" (UniqueName: \"kubernetes.io/projected/8462659d-7d75-4d56-b4c1-75dfc4aea577-kube-api-access-wnj7l\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.553951 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552464    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/10d63418-937e-4dee-a71f-592cfdff55ca-agent-certs\") pod \"konnectivity-agent-2t758\" (UID: \"10d63418-937e-4dee-a71f-592cfdff55ca\") " pod="kube-system/konnectivity-agent-2t758"
Sep 17 18:29:27.553951 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552458    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-run-openvswitch\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.553951 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552489    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a7f6c88f-5861-4089-a4a8-75bb85c16c98-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kpswk\" (UID: \"a7f6c88f-5861-4089-a4a8-75bb85c16c98\") " pod="openshift-multus/multus-additional-cni-plugins-kpswk"
Sep 17 18:29:27.553951 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552491    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mbkj\" (UniqueName: \"kubernetes.io/projected/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-kube-api-access-4mbkj\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.553951 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552514    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-host-run-multus-certs\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.553951 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552566    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-var-lib-openvswitch\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.553951 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552597    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-var-lib-kubelet\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.553951 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552624    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-host-cni-netd\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.553951 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552631    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-var-lib-openvswitch\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.553951 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552587    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f6048ce0-9af4-4205-baa1-c28be5018d95-sys-fs\") pod \"aws-ebs-csi-driver-node-2kh6l\" (UID: \"f6048ce0-9af4-4205-baa1-c28be5018d95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kh6l"
Sep 17 18:29:27.553951 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552650    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.553951 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552682    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c69432b1-9a81-4293-bd1c-4df1df813ba4-ovnkube-config\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.553951 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552691    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-var-lib-kubelet\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.554764 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552700    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/10d63418-937e-4dee-a71f-592cfdff55ca-konnectivity-ca\") pod \"konnectivity-agent-2t758\" (UID: \"10d63418-937e-4dee-a71f-592cfdff55ca\") " pod="kube-system/konnectivity-agent-2t758"
Sep 17 18:29:27.554764 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552707    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c554cd64-2695-47d3-999a-2d8bf8d7ef63-kubelet-config\") pod \"global-pull-secret-syncer-fs688\" (UID: \"c554cd64-2695-47d3-999a-2d8bf8d7ef63\") " pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:29:27.554764 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552730    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-run\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.554764 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552738    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-host-cni-netd\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.554764 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552754    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f6048ce0-9af4-4205-baa1-c28be5018d95-socket-dir\") pod \"aws-ebs-csi-driver-node-2kh6l\" (UID: \"f6048ce0-9af4-4205-baa1-c28be5018d95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kh6l"
Sep 17 18:29:27.554764 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552775    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.554764 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552781    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-etc-openvswitch\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.554764 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552818    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-log-socket\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.554764 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552825    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-etc-openvswitch\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.554764 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552844    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-host-run-ovn-kubernetes\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.554764 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552868    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c554cd64-2695-47d3-999a-2d8bf8d7ef63-dbus\") pod \"global-pull-secret-syncer-fs688\" (UID: \"c554cd64-2695-47d3-999a-2d8bf8d7ef63\") " pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:29:27.554764 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552891    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a7f6c88f-5861-4089-a4a8-75bb85c16c98-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kpswk\" (UID: \"a7f6c88f-5861-4089-a4a8-75bb85c16c98\") " pod="openshift-multus/multus-additional-cni-plugins-kpswk"
Sep 17 18:29:27.554764 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552914    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7969c2b7-d790-4dbd-bb84-4886d90f38d7-iptables-alerter-script\") pod \"iptables-alerter-qsksv\" (UID: \"7969c2b7-d790-4dbd-bb84-4886d90f38d7\") " pod="openshift-network-operator/iptables-alerter-qsksv"
Sep 17 18:29:27.554764 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552937    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-host-kubelet\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.554764 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552961    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-host-cni-bin\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.554764 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.552985    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zwcs\" (UniqueName: \"kubernetes.io/projected/947781e1-4423-435e-b44f-53ca352f8df7-kube-api-access-4zwcs\") pod \"node-ca-5cb86\" (UID: \"947781e1-4423-435e-b44f-53ca352f8df7\") " pod="openshift-image-registry/node-ca-5cb86"
Sep 17 18:29:27.554764 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.553012    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcwlc\" (UniqueName: \"kubernetes.io/projected/9530dc1d-51a9-4c20-aa0f-dbfbf912e02e-kube-api-access-zcwlc\") pod \"network-check-target-cxndc\" (UID: \"9530dc1d-51a9-4c20-aa0f-dbfbf912e02e\") " pod="openshift-network-diagnostics/network-check-target-cxndc"
Sep 17 18:29:27.555519 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.553037    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-systemd-units\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.555519 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.553060    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-run-systemd\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.555519 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.553085    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-host-var-lib-kubelet\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.555519 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.553557    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-etc-sysctl-conf\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.555519 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.553629    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/61513726-d6fb-4e35-a39f-818bcb1a9596-hosts-file\") pod \"node-resolver-745b6\" (UID: \"61513726-d6fb-4e35-a39f-818bcb1a9596\") " pod="openshift-dns/node-resolver-745b6"
Sep 17 18:29:27.555519 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.553681    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c69432b1-9a81-4293-bd1c-4df1df813ba4-env-overrides\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.555519 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.553715    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-multus-cni-dir\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.555519 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.553745    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-multus-conf-dir\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.555519 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.553774    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/947781e1-4423-435e-b44f-53ca352f8df7-host\") pod \"node-ca-5cb86\" (UID: \"947781e1-4423-435e-b44f-53ca352f8df7\") " pod="openshift-image-registry/node-ca-5cb86"
Sep 17 18:29:27.555519 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.553800    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a7f6c88f-5861-4089-a4a8-75bb85c16c98-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kpswk\" (UID: \"a7f6c88f-5861-4089-a4a8-75bb85c16c98\") " pod="openshift-multus/multus-additional-cni-plugins-kpswk"
Sep 17 18:29:27.555519 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.553833    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skqcl\" (UniqueName: \"kubernetes.io/projected/61513726-d6fb-4e35-a39f-818bcb1a9596-kube-api-access-skqcl\") pod \"node-resolver-745b6\" (UID: \"61513726-d6fb-4e35-a39f-818bcb1a9596\") " pod="openshift-dns/node-resolver-745b6"
Sep 17 18:29:27.555519 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.553868    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5skqc\" (UniqueName: \"kubernetes.io/projected/f6048ce0-9af4-4205-baa1-c28be5018d95-kube-api-access-5skqc\") pod \"aws-ebs-csi-driver-node-2kh6l\" (UID: \"f6048ce0-9af4-4205-baa1-c28be5018d95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kh6l"
Sep 17 18:29:27.555519 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.553900    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-host-run-netns\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.555519 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.553933    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-multus-socket-dir-parent\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.555519 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.553960    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-host-var-lib-cni-bin\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.555519 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.554022    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c554cd64-2695-47d3-999a-2d8bf8d7ef63-original-pull-secret\") pod \"global-pull-secret-syncer-fs688\" (UID: \"c554cd64-2695-47d3-999a-2d8bf8d7ef63\") " pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:29:27.555519 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.554053    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a7f6c88f-5861-4089-a4a8-75bb85c16c98-cnibin\") pod \"multus-additional-cni-plugins-kpswk\" (UID: \"a7f6c88f-5861-4089-a4a8-75bb85c16c98\") " pod="openshift-multus/multus-additional-cni-plugins-kpswk"
Sep 17 18:29:27.556302 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.554159    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a7f6c88f-5861-4089-a4a8-75bb85c16c98-cnibin\") pod \"multus-additional-cni-plugins-kpswk\" (UID: \"a7f6c88f-5861-4089-a4a8-75bb85c16c98\") " pod="openshift-multus/multus-additional-cni-plugins-kpswk"
Sep 17 18:29:27.556302 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.554210    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-log-socket\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.556302 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.554262    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-host-run-ovn-kubernetes\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.556302 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.554395    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c554cd64-2695-47d3-999a-2d8bf8d7ef63-dbus\") pod \"global-pull-secret-syncer-fs688\" (UID: \"c554cd64-2695-47d3-999a-2d8bf8d7ef63\") " pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:29:27.556302 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.554912    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a7f6c88f-5861-4089-a4a8-75bb85c16c98-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kpswk\" (UID: \"a7f6c88f-5861-4089-a4a8-75bb85c16c98\") " pod="openshift-multus/multus-additional-cni-plugins-kpswk"
Sep 17 18:29:27.556302 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.555034    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-host-kubelet\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.556302 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.555078    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-host-cni-bin\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.556302 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.555336    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-hostroot\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.556302 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.555410    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-etc-systemd\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.556302 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.555433    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-host\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.556302 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.555567    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c69432b1-9a81-4293-bd1c-4df1df813ba4-ovnkube-config\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.556302 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.555645    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-systemd-units\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.556302 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.555696    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-tmp\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.556302 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.555706    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-run-systemd\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.556302 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.555756    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c554cd64-2695-47d3-999a-2d8bf8d7ef63-kubelet-config\") pod \"global-pull-secret-syncer-fs688\" (UID: \"c554cd64-2695-47d3-999a-2d8bf8d7ef63\") " pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:29:27.556302 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.555833    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f6048ce0-9af4-4205-baa1-c28be5018d95-socket-dir\") pod \"aws-ebs-csi-driver-node-2kh6l\" (UID: \"f6048ce0-9af4-4205-baa1-c28be5018d95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kh6l"
Sep 17 18:29:27.556302 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.555912    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/10d63418-937e-4dee-a71f-592cfdff55ca-agent-certs\") pod \"konnectivity-agent-2t758\" (UID: \"10d63418-937e-4dee-a71f-592cfdff55ca\") " pod="kube-system/konnectivity-agent-2t758"
Sep 17 18:29:27.556302 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.555990    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c69432b1-9a81-4293-bd1c-4df1df813ba4-ovn-node-metrics-cert\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.557186 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.556014    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-multus-conf-dir\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.557186 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.556089    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-multus-cni-dir\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.557186 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.556126    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-etc-sysctl-conf\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.557186 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.556140    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-host-var-lib-kubelet\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.557186 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.556141    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7969c2b7-d790-4dbd-bb84-4886d90f38d7-iptables-alerter-script\") pod \"iptables-alerter-qsksv\" (UID: \"7969c2b7-d790-4dbd-bb84-4886d90f38d7\") " pod="openshift-network-operator/iptables-alerter-qsksv"
Sep 17 18:29:27.557186 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.556193    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/61513726-d6fb-4e35-a39f-818bcb1a9596-hosts-file\") pod \"node-resolver-745b6\" (UID: \"61513726-d6fb-4e35-a39f-818bcb1a9596\") " pod="openshift-dns/node-resolver-745b6"
Sep 17 18:29:27.557186 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.556239    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/947781e1-4423-435e-b44f-53ca352f8df7-host\") pod \"node-ca-5cb86\" (UID: \"947781e1-4423-435e-b44f-53ca352f8df7\") " pod="openshift-image-registry/node-ca-5cb86"
Sep 17 18:29:27.557186 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.556327    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a7f6c88f-5861-4089-a4a8-75bb85c16c98-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kpswk\" (UID: \"a7f6c88f-5861-4089-a4a8-75bb85c16c98\") " pod="openshift-multus/multus-additional-cni-plugins-kpswk"
Sep 17 18:29:27.557186 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.556454    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-multus-socket-dir-parent\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.557186 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.556509    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c69432b1-9a81-4293-bd1c-4df1df813ba4-host-run-netns\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.557186 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.556559    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8462659d-7d75-4d56-b4c1-75dfc4aea577-host-var-lib-cni-bin\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.557186 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.556592    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c69432b1-9a81-4293-bd1c-4df1df813ba4-env-overrides\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.557186 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:27.556610    2588 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:27.557186 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.556650    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-run\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.557186 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:27.556789    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c554cd64-2695-47d3-999a-2d8bf8d7ef63-original-pull-secret podName:c554cd64-2695-47d3-999a-2d8bf8d7ef63 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:28.056768627 +0000 UTC m=+4.907774008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c554cd64-2695-47d3-999a-2d8bf8d7ef63-original-pull-secret") pod "global-pull-secret-syncer-fs688" (UID: "c554cd64-2695-47d3-999a-2d8bf8d7ef63") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:27.557186 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.556936    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c69432b1-9a81-4293-bd1c-4df1df813ba4-ovnkube-script-lib\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.557958 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.557838    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-etc-tuned\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.560503 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.560480    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w6nh\" (UniqueName: \"kubernetes.io/projected/a7f6c88f-5861-4089-a4a8-75bb85c16c98-kube-api-access-8w6nh\") pod \"multus-additional-cni-plugins-kpswk\" (UID: \"a7f6c88f-5861-4089-a4a8-75bb85c16c98\") " pod="openshift-multus/multus-additional-cni-plugins-kpswk"
Sep 17 18:29:27.562969 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.562946    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk9wx\" (UniqueName: \"kubernetes.io/projected/1c01ccd8-9e9c-467f-a2af-50720f272038-kube-api-access-fk9wx\") pod \"network-metrics-daemon-bnq8f\" (UID: \"1c01ccd8-9e9c-467f-a2af-50720f272038\") " pod="openshift-multus/network-metrics-daemon-bnq8f"
Sep 17 18:29:27.563937 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:27.563911    2588 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:27.564050 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:27.563942    2588 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:27.564050 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:27.563957    2588 projected.go:196] Error preparing data for projected volume kube-api-access-zcwlc for pod openshift-network-diagnostics/network-check-target-cxndc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:27.564050 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:27.564011    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9530dc1d-51a9-4c20-aa0f-dbfbf912e02e-kube-api-access-zcwlc podName:9530dc1d-51a9-4c20-aa0f-dbfbf912e02e nodeName:}" failed. No retries permitted until 2026-09-17 18:29:28.063994691 +0000 UTC m=+4.915000085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zcwlc" (UniqueName: "kubernetes.io/projected/9530dc1d-51a9-4c20-aa0f-dbfbf912e02e-kube-api-access-zcwlc") pod "network-check-target-cxndc" (UID: "9530dc1d-51a9-4c20-aa0f-dbfbf912e02e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:27.565406 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.565380    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mbkj\" (UniqueName: \"kubernetes.io/projected/9254a7f2-3a79-421d-be0c-2cb1fbfe46fa-kube-api-access-4mbkj\") pod \"tuned-z97dh\" (UID: \"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa\") " pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.566227 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.566199    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnj7l\" (UniqueName: \"kubernetes.io/projected/8462659d-7d75-4d56-b4c1-75dfc4aea577-kube-api-access-wnj7l\") pod \"multus-f96pg\" (UID: \"8462659d-7d75-4d56-b4c1-75dfc4aea577\") " pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.566651 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.566630    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skqcl\" (UniqueName: \"kubernetes.io/projected/61513726-d6fb-4e35-a39f-818bcb1a9596-kube-api-access-skqcl\") pod \"node-resolver-745b6\" (UID: \"61513726-d6fb-4e35-a39f-818bcb1a9596\") " pod="openshift-dns/node-resolver-745b6"
Sep 17 18:29:27.566806 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.566698    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhp2j\" (UniqueName: \"kubernetes.io/projected/7969c2b7-d790-4dbd-bb84-4886d90f38d7-kube-api-access-dhp2j\") pod \"iptables-alerter-qsksv\" (UID: \"7969c2b7-d790-4dbd-bb84-4886d90f38d7\") " pod="openshift-network-operator/iptables-alerter-qsksv"
Sep 17 18:29:27.566806 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.566701    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zwcs\" (UniqueName: \"kubernetes.io/projected/947781e1-4423-435e-b44f-53ca352f8df7-kube-api-access-4zwcs\") pod \"node-ca-5cb86\" (UID: \"947781e1-4423-435e-b44f-53ca352f8df7\") " pod="openshift-image-registry/node-ca-5cb86"
Sep 17 18:29:27.567281 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.567258    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5skqc\" (UniqueName: \"kubernetes.io/projected/f6048ce0-9af4-4205-baa1-c28be5018d95-kube-api-access-5skqc\") pod \"aws-ebs-csi-driver-node-2kh6l\" (UID: \"f6048ce0-9af4-4205-baa1-c28be5018d95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kh6l"
Sep 17 18:29:27.567434 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.567417    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks65r\" (UniqueName: \"kubernetes.io/projected/c69432b1-9a81-4293-bd1c-4df1df813ba4-kube-api-access-ks65r\") pod \"ovnkube-node-2rqhq\" (UID: \"c69432b1-9a81-4293-bd1c-4df1df813ba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.707991 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.707903    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qsksv"
Sep 17 18:29:27.719984 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.719954    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z97dh"
Sep 17 18:29:27.728196 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.728175    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-745b6"
Sep 17 18:29:27.735698 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.735680    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-f96pg"
Sep 17 18:29:27.741617 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.741600    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kh6l"
Sep 17 18:29:27.748366 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.748342    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2t758"
Sep 17 18:29:27.755336 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.755313    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:27.762081 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.762063    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5cb86"
Sep 17 18:29:27.767512 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:27.767494    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kpswk"
Sep 17 18:29:28.057081 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:28.056994    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c554cd64-2695-47d3-999a-2d8bf8d7ef63-original-pull-secret\") pod \"global-pull-secret-syncer-fs688\" (UID: \"c554cd64-2695-47d3-999a-2d8bf8d7ef63\") " pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:29:28.057081 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:28.057036    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c01ccd8-9e9c-467f-a2af-50720f272038-metrics-certs\") pod \"network-metrics-daemon-bnq8f\" (UID: \"1c01ccd8-9e9c-467f-a2af-50720f272038\") " pod="openshift-multus/network-metrics-daemon-bnq8f"
Sep 17 18:29:28.057301 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:28.057161    2588 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:28.057301 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:28.057240    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c554cd64-2695-47d3-999a-2d8bf8d7ef63-original-pull-secret podName:c554cd64-2695-47d3-999a-2d8bf8d7ef63 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:29.057218664 +0000 UTC m=+5.908224061 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c554cd64-2695-47d3-999a-2d8bf8d7ef63-original-pull-secret") pod "global-pull-secret-syncer-fs688" (UID: "c554cd64-2695-47d3-999a-2d8bf8d7ef63") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:28.057301 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:28.057161    2588 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:28.057454 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:28.057317    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c01ccd8-9e9c-467f-a2af-50720f272038-metrics-certs podName:1c01ccd8-9e9c-467f-a2af-50720f272038 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:29.057299085 +0000 UTC m=+5.908304464 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c01ccd8-9e9c-467f-a2af-50720f272038-metrics-certs") pod "network-metrics-daemon-bnq8f" (UID: "1c01ccd8-9e9c-467f-a2af-50720f272038") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:28.158381 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:28.158337    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcwlc\" (UniqueName: \"kubernetes.io/projected/9530dc1d-51a9-4c20-aa0f-dbfbf912e02e-kube-api-access-zcwlc\") pod \"network-check-target-cxndc\" (UID: \"9530dc1d-51a9-4c20-aa0f-dbfbf912e02e\") " pod="openshift-network-diagnostics/network-check-target-cxndc"
Sep 17 18:29:28.158588 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:28.158563    2588 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:28.158661 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:28.158594    2588 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:28.158661 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:28.158605    2588 projected.go:196] Error preparing data for projected volume kube-api-access-zcwlc for pod openshift-network-diagnostics/network-check-target-cxndc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:28.158755 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:28.158671    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9530dc1d-51a9-4c20-aa0f-dbfbf912e02e-kube-api-access-zcwlc podName:9530dc1d-51a9-4c20-aa0f-dbfbf912e02e nodeName:}" failed. No retries permitted until 2026-09-17 18:29:29.158651599 +0000 UTC m=+6.009656995 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zcwlc" (UniqueName: "kubernetes.io/projected/9530dc1d-51a9-4c20-aa0f-dbfbf912e02e-kube-api-access-zcwlc") pod "network-check-target-cxndc" (UID: "9530dc1d-51a9-4c20-aa0f-dbfbf912e02e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:28.268736 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:28.268688    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f6c88f_5861_4089_a4a8_75bb85c16c98.slice/crio-48d0dc6cac502387d151e61fc00314de3da702190b46e150193157234ddf09f2 WatchSource:0}: Error finding container 48d0dc6cac502387d151e61fc00314de3da702190b46e150193157234ddf09f2: Status 404 returned error can't find the container with id 48d0dc6cac502387d151e61fc00314de3da702190b46e150193157234ddf09f2
Sep 17 18:29:28.269373 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:28.269349    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8462659d_7d75_4d56_b4c1_75dfc4aea577.slice/crio-99fb0eee41b9354b3f1fbc381880f3260306d0e916fb20ecce9a90061f4eded1 WatchSource:0}: Error finding container 99fb0eee41b9354b3f1fbc381880f3260306d0e916fb20ecce9a90061f4eded1: Status 404 returned error can't find the container with id 99fb0eee41b9354b3f1fbc381880f3260306d0e916fb20ecce9a90061f4eded1
Sep 17 18:29:28.270836 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:28.270674    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10d63418_937e_4dee_a71f_592cfdff55ca.slice/crio-a389e6c555237aa5c3b27e793fd959500170cebc2bdfb39f072febd98e304a2b WatchSource:0}: Error finding container a389e6c555237aa5c3b27e793fd959500170cebc2bdfb39f072febd98e304a2b: Status 404 returned error can't find the container with id a389e6c555237aa5c3b27e793fd959500170cebc2bdfb39f072febd98e304a2b
Sep 17 18:29:28.272131 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:28.272034    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod947781e1_4423_435e_b44f_53ca352f8df7.slice/crio-e537ccaac8152ca504b9a4174408c5843364742c34b9b994239dd67d2801351f WatchSource:0}: Error finding container e537ccaac8152ca504b9a4174408c5843364742c34b9b994239dd67d2801351f: Status 404 returned error can't find the container with id e537ccaac8152ca504b9a4174408c5843364742c34b9b994239dd67d2801351f
Sep 17 18:29:28.274499 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:28.274464    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc69432b1_9a81_4293_bd1c_4df1df813ba4.slice/crio-4f38a6a05b6dc4dd4b1ba2e5e3e7a67ed53919aa6b8b45a0f4efda54b5b07b57 WatchSource:0}: Error finding container 4f38a6a05b6dc4dd4b1ba2e5e3e7a67ed53919aa6b8b45a0f4efda54b5b07b57: Status 404 returned error can't find the container with id 4f38a6a05b6dc4dd4b1ba2e5e3e7a67ed53919aa6b8b45a0f4efda54b5b07b57
Sep 17 18:29:28.276386 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:28.276302    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7969c2b7_d790_4dbd_bb84_4886d90f38d7.slice/crio-a4bbbfa843e2373a99f4aafc49f7cfe6d3f9c6f0c9d93e6870334b9519b698ef WatchSource:0}: Error finding container a4bbbfa843e2373a99f4aafc49f7cfe6d3f9c6f0c9d93e6870334b9519b698ef: Status 404 returned error can't find the container with id a4bbbfa843e2373a99f4aafc49f7cfe6d3f9c6f0c9d93e6870334b9519b698ef
Sep 17 18:29:28.277165 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:28.277125    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6048ce0_9af4_4205_baa1_c28be5018d95.slice/crio-cd7e3fcaab2c7b27e22217464c19a92ea2fd07a0718df46545809140f9e328ca WatchSource:0}: Error finding container cd7e3fcaab2c7b27e22217464c19a92ea2fd07a0718df46545809140f9e328ca: Status 404 returned error can't find the container with id cd7e3fcaab2c7b27e22217464c19a92ea2fd07a0718df46545809140f9e328ca
Sep 17 18:29:28.278725 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:28.278007    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61513726_d6fb_4e35_a39f_818bcb1a9596.slice/crio-966eb8faacbd2e17764cd1740ca7f48b880e2291afc5f2d5f0a7c3e6997c6dd3 WatchSource:0}: Error finding container 966eb8faacbd2e17764cd1740ca7f48b880e2291afc5f2d5f0a7c3e6997c6dd3: Status 404 returned error can't find the container with id 966eb8faacbd2e17764cd1740ca7f48b880e2291afc5f2d5f0a7c3e6997c6dd3
Sep 17 18:29:28.279519 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:29:28.279497    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9254a7f2_3a79_421d_be0c_2cb1fbfe46fa.slice/crio-ae0d0ea0340bc9ca4b1aca96b028f82e89667fae38e34d002783d9dae84c8c82 WatchSource:0}: Error finding container ae0d0ea0340bc9ca4b1aca96b028f82e89667fae38e34d002783d9dae84c8c82: Status 404 returned error can't find the container with id ae0d0ea0340bc9ca4b1aca96b028f82e89667fae38e34d002783d9dae84c8c82
Sep 17 18:29:28.482499 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:28.482466    2588 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 18:24:27 +0000 UTC" deadline="2028-04-24 14:25:08.766696678 +0000 UTC"
Sep 17 18:29:28.482499 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:28.482494    2588 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14035h55m40.284206582s"
Sep 17 18:29:28.509447 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:28.509207    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z97dh" event={"ID":"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa","Type":"ContainerStarted","Data":"ae0d0ea0340bc9ca4b1aca96b028f82e89667fae38e34d002783d9dae84c8c82"}
Sep 17 18:29:28.510315 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:28.510293    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-745b6" event={"ID":"61513726-d6fb-4e35-a39f-818bcb1a9596","Type":"ContainerStarted","Data":"966eb8faacbd2e17764cd1740ca7f48b880e2291afc5f2d5f0a7c3e6997c6dd3"}
Sep 17 18:29:28.511482 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:28.511453    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kh6l" event={"ID":"f6048ce0-9af4-4205-baa1-c28be5018d95","Type":"ContainerStarted","Data":"cd7e3fcaab2c7b27e22217464c19a92ea2fd07a0718df46545809140f9e328ca"}
Sep 17 18:29:28.512420 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:28.512398    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qsksv" event={"ID":"7969c2b7-d790-4dbd-bb84-4886d90f38d7","Type":"ContainerStarted","Data":"a4bbbfa843e2373a99f4aafc49f7cfe6d3f9c6f0c9d93e6870334b9519b698ef"}
Sep 17 18:29:28.513356 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:28.513339    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq" event={"ID":"c69432b1-9a81-4293-bd1c-4df1df813ba4","Type":"ContainerStarted","Data":"4f38a6a05b6dc4dd4b1ba2e5e3e7a67ed53919aa6b8b45a0f4efda54b5b07b57"}
Sep 17 18:29:28.514461 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:28.514440    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5cb86" event={"ID":"947781e1-4423-435e-b44f-53ca352f8df7","Type":"ContainerStarted","Data":"e537ccaac8152ca504b9a4174408c5843364742c34b9b994239dd67d2801351f"}
Sep 17 18:29:28.515491 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:28.515470    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f96pg" event={"ID":"8462659d-7d75-4d56-b4c1-75dfc4aea577","Type":"ContainerStarted","Data":"99fb0eee41b9354b3f1fbc381880f3260306d0e916fb20ecce9a90061f4eded1"}
Sep 17 18:29:28.517100 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:28.517078    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-132.ec2.internal" event={"ID":"189465409bf7b6f4cb62d3daabaf12f1","Type":"ContainerStarted","Data":"eb5acedbc3c85ac0081c2634c263ff0a206df36f55d4e2e78616084a9bdd7a8a"}
Sep 17 18:29:28.520084 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:28.520066    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2t758" event={"ID":"10d63418-937e-4dee-a71f-592cfdff55ca","Type":"ContainerStarted","Data":"a389e6c555237aa5c3b27e793fd959500170cebc2bdfb39f072febd98e304a2b"}
Sep 17 18:29:28.521098 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:28.521079    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kpswk" event={"ID":"a7f6c88f-5861-4089-a4a8-75bb85c16c98","Type":"ContainerStarted","Data":"48d0dc6cac502387d151e61fc00314de3da702190b46e150193157234ddf09f2"}
Sep 17 18:29:28.534958 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:28.534914    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-132.ec2.internal" podStartSLOduration=1.53489981 podStartE2EDuration="1.53489981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:28.534641441 +0000 UTC m=+5.385646840" watchObservedRunningTime="2026-09-17 18:29:28.53489981 +0000 UTC m=+5.385905208"
Sep 17 18:29:29.066362 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:29.065570    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c01ccd8-9e9c-467f-a2af-50720f272038-metrics-certs\") pod \"network-metrics-daemon-bnq8f\" (UID: \"1c01ccd8-9e9c-467f-a2af-50720f272038\") " pod="openshift-multus/network-metrics-daemon-bnq8f"
Sep 17 18:29:29.066362 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:29.065677    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c554cd64-2695-47d3-999a-2d8bf8d7ef63-original-pull-secret\") pod \"global-pull-secret-syncer-fs688\" (UID: \"c554cd64-2695-47d3-999a-2d8bf8d7ef63\") " pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:29:29.066362 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:29.065794    2588 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:29.066362 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:29.065861    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c554cd64-2695-47d3-999a-2d8bf8d7ef63-original-pull-secret podName:c554cd64-2695-47d3-999a-2d8bf8d7ef63 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:31.065839381 +0000 UTC m=+7.916844780 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c554cd64-2695-47d3-999a-2d8bf8d7ef63-original-pull-secret") pod "global-pull-secret-syncer-fs688" (UID: "c554cd64-2695-47d3-999a-2d8bf8d7ef63") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:29.066362 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:29.066272    2588 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:29.066362 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:29.066327    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c01ccd8-9e9c-467f-a2af-50720f272038-metrics-certs podName:1c01ccd8-9e9c-467f-a2af-50720f272038 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:31.066310558 +0000 UTC m=+7.917315940 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c01ccd8-9e9c-467f-a2af-50720f272038-metrics-certs") pod "network-metrics-daemon-bnq8f" (UID: "1c01ccd8-9e9c-467f-a2af-50720f272038") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:29.166901 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:29.166860    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcwlc\" (UniqueName: \"kubernetes.io/projected/9530dc1d-51a9-4c20-aa0f-dbfbf912e02e-kube-api-access-zcwlc\") pod \"network-check-target-cxndc\" (UID: \"9530dc1d-51a9-4c20-aa0f-dbfbf912e02e\") " pod="openshift-network-diagnostics/network-check-target-cxndc"
Sep 17 18:29:29.167088 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:29.167047    2588 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:29.167088 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:29.167073    2588 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:29.167088 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:29.167087    2588 projected.go:196] Error preparing data for projected volume kube-api-access-zcwlc for pod openshift-network-diagnostics/network-check-target-cxndc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:29.167254 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:29.167148    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9530dc1d-51a9-4c20-aa0f-dbfbf912e02e-kube-api-access-zcwlc podName:9530dc1d-51a9-4c20-aa0f-dbfbf912e02e nodeName:}" failed. No retries permitted until 2026-09-17 18:29:31.167128701 +0000 UTC m=+8.018134083 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zcwlc" (UniqueName: "kubernetes.io/projected/9530dc1d-51a9-4c20-aa0f-dbfbf912e02e-kube-api-access-zcwlc") pod "network-check-target-cxndc" (UID: "9530dc1d-51a9-4c20-aa0f-dbfbf912e02e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:29.483749 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:29.483596    2588 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 18:24:27 +0000 UTC" deadline="2028-06-04 15:42:07.590002192 +0000 UTC"
Sep 17 18:29:29.483749 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:29.483657    2588 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15021h12m38.106348807s"
Sep 17 18:29:29.503344 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:29.502647    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:29:29.503344 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:29.502775    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs688" podUID="c554cd64-2695-47d3-999a-2d8bf8d7ef63"
Sep 17 18:29:29.503344 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:29.503177    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bnq8f"
Sep 17 18:29:29.503344 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:29.503277    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bnq8f" podUID="1c01ccd8-9e9c-467f-a2af-50720f272038"
Sep 17 18:29:29.503975 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:29.503845    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxndc"
Sep 17 18:29:29.503975 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:29.503939    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxndc" podUID="9530dc1d-51a9-4c20-aa0f-dbfbf912e02e"
Sep 17 18:29:29.532435 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:29.531843    2588 generic.go:356] "Generic (PLEG): container finished" podID="8d590791640aecc72b49fc406401ef47" containerID="1a1e153ed320782819a10c84868e6411293109d3758aab4044069c12e474510d" exitCode=0
Sep 17 18:29:29.532435 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:29.532200    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-132.ec2.internal" event={"ID":"8d590791640aecc72b49fc406401ef47","Type":"ContainerDied","Data":"1a1e153ed320782819a10c84868e6411293109d3758aab4044069c12e474510d"}
Sep 17 18:29:30.559291 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:30.559251    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-132.ec2.internal" event={"ID":"8d590791640aecc72b49fc406401ef47","Type":"ContainerStarted","Data":"e112789d89153bb06d8b77df71643a6d48eb3be4446d677b71003ce37102f5ea"}
Sep 17 18:29:31.085107 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:31.084278    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c554cd64-2695-47d3-999a-2d8bf8d7ef63-original-pull-secret\") pod \"global-pull-secret-syncer-fs688\" (UID: \"c554cd64-2695-47d3-999a-2d8bf8d7ef63\") " pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:29:31.085107 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:31.084330    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c01ccd8-9e9c-467f-a2af-50720f272038-metrics-certs\") pod \"network-metrics-daemon-bnq8f\" (UID: \"1c01ccd8-9e9c-467f-a2af-50720f272038\") " pod="openshift-multus/network-metrics-daemon-bnq8f"
Sep 17 18:29:31.085107 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:31.084478    2588 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:31.085107 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:31.084553    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c01ccd8-9e9c-467f-a2af-50720f272038-metrics-certs podName:1c01ccd8-9e9c-467f-a2af-50720f272038 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:35.084520705 +0000 UTC m=+11.935526096 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c01ccd8-9e9c-467f-a2af-50720f272038-metrics-certs") pod "network-metrics-daemon-bnq8f" (UID: "1c01ccd8-9e9c-467f-a2af-50720f272038") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:31.085107 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:31.084968    2588 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:31.085107 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:31.085020    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c554cd64-2695-47d3-999a-2d8bf8d7ef63-original-pull-secret podName:c554cd64-2695-47d3-999a-2d8bf8d7ef63 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:35.085004002 +0000 UTC m=+11.936009383 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c554cd64-2695-47d3-999a-2d8bf8d7ef63-original-pull-secret") pod "global-pull-secret-syncer-fs688" (UID: "c554cd64-2695-47d3-999a-2d8bf8d7ef63") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:31.186306 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:31.185624    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcwlc\" (UniqueName: \"kubernetes.io/projected/9530dc1d-51a9-4c20-aa0f-dbfbf912e02e-kube-api-access-zcwlc\") pod \"network-check-target-cxndc\" (UID: \"9530dc1d-51a9-4c20-aa0f-dbfbf912e02e\") " pod="openshift-network-diagnostics/network-check-target-cxndc"
Sep 17 18:29:31.186306 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:31.185814    2588 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:31.186306 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:31.185837    2588 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:31.186306 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:31.185849    2588 projected.go:196] Error preparing data for projected volume kube-api-access-zcwlc for pod openshift-network-diagnostics/network-check-target-cxndc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:31.186306 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:31.185907    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9530dc1d-51a9-4c20-aa0f-dbfbf912e02e-kube-api-access-zcwlc podName:9530dc1d-51a9-4c20-aa0f-dbfbf912e02e nodeName:}" failed. No retries permitted until 2026-09-17 18:29:35.185887823 +0000 UTC m=+12.036893213 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zcwlc" (UniqueName: "kubernetes.io/projected/9530dc1d-51a9-4c20-aa0f-dbfbf912e02e-kube-api-access-zcwlc") pod "network-check-target-cxndc" (UID: "9530dc1d-51a9-4c20-aa0f-dbfbf912e02e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:31.501956 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:31.501638    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bnq8f"
Sep 17 18:29:31.501956 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:31.501656    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxndc"
Sep 17 18:29:31.501956 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:31.501638    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:29:31.501956 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:31.501795    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bnq8f" podUID="1c01ccd8-9e9c-467f-a2af-50720f272038"
Sep 17 18:29:31.501956 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:31.501941    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs688" podUID="c554cd64-2695-47d3-999a-2d8bf8d7ef63"
Sep 17 18:29:31.502371 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:31.502027    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxndc" podUID="9530dc1d-51a9-4c20-aa0f-dbfbf912e02e"
Sep 17 18:29:33.500869 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:33.500833    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxndc"
Sep 17 18:29:33.501333 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:33.500966    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxndc" podUID="9530dc1d-51a9-4c20-aa0f-dbfbf912e02e"
Sep 17 18:29:33.501460 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:33.501432    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:29:33.501563 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:33.501542    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs688" podUID="c554cd64-2695-47d3-999a-2d8bf8d7ef63"
Sep 17 18:29:33.501635 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:33.501622    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bnq8f"
Sep 17 18:29:33.501759 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:33.501722    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bnq8f" podUID="1c01ccd8-9e9c-467f-a2af-50720f272038"
Sep 17 18:29:35.120971 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:35.120929    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c554cd64-2695-47d3-999a-2d8bf8d7ef63-original-pull-secret\") pod \"global-pull-secret-syncer-fs688\" (UID: \"c554cd64-2695-47d3-999a-2d8bf8d7ef63\") " pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:29:35.121421 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:35.120981    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c01ccd8-9e9c-467f-a2af-50720f272038-metrics-certs\") pod \"network-metrics-daemon-bnq8f\" (UID: \"1c01ccd8-9e9c-467f-a2af-50720f272038\") " pod="openshift-multus/network-metrics-daemon-bnq8f"
Sep 17 18:29:35.121421 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:35.121110    2588 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:35.121421 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:35.121180    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c554cd64-2695-47d3-999a-2d8bf8d7ef63-original-pull-secret podName:c554cd64-2695-47d3-999a-2d8bf8d7ef63 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:43.121160678 +0000 UTC m=+19.972166100 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c554cd64-2695-47d3-999a-2d8bf8d7ef63-original-pull-secret") pod "global-pull-secret-syncer-fs688" (UID: "c554cd64-2695-47d3-999a-2d8bf8d7ef63") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:35.121421 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:35.121113    2588 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:35.121421 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:35.121342    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c01ccd8-9e9c-467f-a2af-50720f272038-metrics-certs podName:1c01ccd8-9e9c-467f-a2af-50720f272038 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:43.121323553 +0000 UTC m=+19.972328944 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c01ccd8-9e9c-467f-a2af-50720f272038-metrics-certs") pod "network-metrics-daemon-bnq8f" (UID: "1c01ccd8-9e9c-467f-a2af-50720f272038") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:35.222309 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:35.222226    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcwlc\" (UniqueName: \"kubernetes.io/projected/9530dc1d-51a9-4c20-aa0f-dbfbf912e02e-kube-api-access-zcwlc\") pod \"network-check-target-cxndc\" (UID: \"9530dc1d-51a9-4c20-aa0f-dbfbf912e02e\") " pod="openshift-network-diagnostics/network-check-target-cxndc"
Sep 17 18:29:35.222492 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:35.222413    2588 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:35.222492 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:35.222435    2588 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:35.222492 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:35.222447    2588 projected.go:196] Error preparing data for projected volume kube-api-access-zcwlc for pod openshift-network-diagnostics/network-check-target-cxndc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:35.222679 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:35.222508    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9530dc1d-51a9-4c20-aa0f-dbfbf912e02e-kube-api-access-zcwlc podName:9530dc1d-51a9-4c20-aa0f-dbfbf912e02e nodeName:}" failed. No retries permitted until 2026-09-17 18:29:43.222487121 +0000 UTC m=+20.073492498 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zcwlc" (UniqueName: "kubernetes.io/projected/9530dc1d-51a9-4c20-aa0f-dbfbf912e02e-kube-api-access-zcwlc") pod "network-check-target-cxndc" (UID: "9530dc1d-51a9-4c20-aa0f-dbfbf912e02e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:35.501563 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:35.501401    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bnq8f"
Sep 17 18:29:35.501563 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:35.501523    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxndc"
Sep 17 18:29:35.501563 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:35.501545    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bnq8f" podUID="1c01ccd8-9e9c-467f-a2af-50720f272038"
Sep 17 18:29:35.501845 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:35.501666    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:29:35.501845 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:35.501699    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxndc" podUID="9530dc1d-51a9-4c20-aa0f-dbfbf912e02e"
Sep 17 18:29:35.501845 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:35.501749    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs688" podUID="c554cd64-2695-47d3-999a-2d8bf8d7ef63"
Sep 17 18:29:37.501235 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:37.501198    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bnq8f"
Sep 17 18:29:37.501790 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:37.501198    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:29:37.501790 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:37.501198    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxndc"
Sep 17 18:29:37.501790 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:37.501354    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bnq8f" podUID="1c01ccd8-9e9c-467f-a2af-50720f272038"
Sep 17 18:29:37.501790 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:37.501427    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs688" podUID="c554cd64-2695-47d3-999a-2d8bf8d7ef63"
Sep 17 18:29:37.501790 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:37.501500    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxndc" podUID="9530dc1d-51a9-4c20-aa0f-dbfbf912e02e"
Sep 17 18:29:39.500699 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:39.500668    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:29:39.501142 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:39.500667    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bnq8f"
Sep 17 18:29:39.501142 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:39.500786    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs688" podUID="c554cd64-2695-47d3-999a-2d8bf8d7ef63"
Sep 17 18:29:39.501142 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:39.500674    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxndc"
Sep 17 18:29:39.501142 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:39.500880    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bnq8f" podUID="1c01ccd8-9e9c-467f-a2af-50720f272038"
Sep 17 18:29:39.501142 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:39.500978    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxndc" podUID="9530dc1d-51a9-4c20-aa0f-dbfbf912e02e"
Sep 17 18:29:41.500715 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:41.500683    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bnq8f"
Sep 17 18:29:41.500715 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:41.500684    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxndc"
Sep 17 18:29:41.501208 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:41.500691    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:29:41.501208 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:41.500854    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxndc" podUID="9530dc1d-51a9-4c20-aa0f-dbfbf912e02e"
Sep 17 18:29:41.501208 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:41.500922    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs688" podUID="c554cd64-2695-47d3-999a-2d8bf8d7ef63"
Sep 17 18:29:41.501208 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:41.501028    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bnq8f" podUID="1c01ccd8-9e9c-467f-a2af-50720f272038"
Sep 17 18:29:43.182167 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:43.182126    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c554cd64-2695-47d3-999a-2d8bf8d7ef63-original-pull-secret\") pod \"global-pull-secret-syncer-fs688\" (UID: \"c554cd64-2695-47d3-999a-2d8bf8d7ef63\") " pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:29:43.182608 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:43.182179    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c01ccd8-9e9c-467f-a2af-50720f272038-metrics-certs\") pod \"network-metrics-daemon-bnq8f\" (UID: \"1c01ccd8-9e9c-467f-a2af-50720f272038\") " pod="openshift-multus/network-metrics-daemon-bnq8f"
Sep 17 18:29:43.182608 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:43.182295    2588 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:43.182608 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:43.182308    2588 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:43.182608 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:43.182362    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c01ccd8-9e9c-467f-a2af-50720f272038-metrics-certs podName:1c01ccd8-9e9c-467f-a2af-50720f272038 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:59.182342479 +0000 UTC m=+36.033347859 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c01ccd8-9e9c-467f-a2af-50720f272038-metrics-certs") pod "network-metrics-daemon-bnq8f" (UID: "1c01ccd8-9e9c-467f-a2af-50720f272038") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:43.182608 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:43.182376    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c554cd64-2695-47d3-999a-2d8bf8d7ef63-original-pull-secret podName:c554cd64-2695-47d3-999a-2d8bf8d7ef63 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:59.182370344 +0000 UTC m=+36.033375720 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c554cd64-2695-47d3-999a-2d8bf8d7ef63-original-pull-secret") pod "global-pull-secret-syncer-fs688" (UID: "c554cd64-2695-47d3-999a-2d8bf8d7ef63") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:43.282829 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:43.282791    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcwlc\" (UniqueName: \"kubernetes.io/projected/9530dc1d-51a9-4c20-aa0f-dbfbf912e02e-kube-api-access-zcwlc\") pod \"network-check-target-cxndc\" (UID: \"9530dc1d-51a9-4c20-aa0f-dbfbf912e02e\") " pod="openshift-network-diagnostics/network-check-target-cxndc"
Sep 17 18:29:43.283020 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:43.282984    2588 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:43.283020 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:43.283011    2588 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:43.283121 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:43.283025    2588 projected.go:196] Error preparing data for projected volume kube-api-access-zcwlc for pod openshift-network-diagnostics/network-check-target-cxndc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:43.283121 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:43.283090    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9530dc1d-51a9-4c20-aa0f-dbfbf912e02e-kube-api-access-zcwlc podName:9530dc1d-51a9-4c20-aa0f-dbfbf912e02e nodeName:}" failed. No retries permitted until 2026-09-17 18:29:59.283070976 +0000 UTC m=+36.134076356 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zcwlc" (UniqueName: "kubernetes.io/projected/9530dc1d-51a9-4c20-aa0f-dbfbf912e02e-kube-api-access-zcwlc") pod "network-check-target-cxndc" (UID: "9530dc1d-51a9-4c20-aa0f-dbfbf912e02e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:43.501032 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:43.500950    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxndc"
Sep 17 18:29:43.501032 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:43.500976    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:29:43.501249 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:43.500946    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bnq8f"
Sep 17 18:29:43.501249 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:43.501079    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxndc" podUID="9530dc1d-51a9-4c20-aa0f-dbfbf912e02e"
Sep 17 18:29:43.501249 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:43.501157    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs688" podUID="c554cd64-2695-47d3-999a-2d8bf8d7ef63"
Sep 17 18:29:43.501365 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:43.501257    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bnq8f" podUID="1c01ccd8-9e9c-467f-a2af-50720f272038"
Sep 17 18:29:45.500918 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:45.500898    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:29:45.501242 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:45.500918    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bnq8f"
Sep 17 18:29:45.501242 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:45.500922    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxndc"
Sep 17 18:29:45.501242 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:45.501021    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs688" podUID="c554cd64-2695-47d3-999a-2d8bf8d7ef63"
Sep 17 18:29:45.501242 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:45.501069    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bnq8f" podUID="1c01ccd8-9e9c-467f-a2af-50720f272038"
Sep 17 18:29:45.501242 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:45.501129    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxndc" podUID="9530dc1d-51a9-4c20-aa0f-dbfbf912e02e"
Sep 17 18:29:46.592817 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:46.592779    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kh6l" event={"ID":"f6048ce0-9af4-4205-baa1-c28be5018d95","Type":"ContainerStarted","Data":"09d530123c0b696531b63d73b0a3c7de09e89a3e6f77defa6c97d5bf53fbeeed"}
Sep 17 18:29:46.596265 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:46.596243    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovn-acl-logging/0.log"
Sep 17 18:29:46.596701 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:46.596671    2588 generic.go:356] "Generic (PLEG): container finished" podID="c69432b1-9a81-4293-bd1c-4df1df813ba4" containerID="70f5e3051ed3c6b7354ea274d2758fd5e61211cba003a43569ebc2e030dd6f73" exitCode=1
Sep 17 18:29:46.596774 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:46.596716    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq" event={"ID":"c69432b1-9a81-4293-bd1c-4df1df813ba4","Type":"ContainerStarted","Data":"2d219962bbb02447ae4fcf60361475f13aa4f3a160e5344782bade34e06a1b96"}
Sep 17 18:29:46.596819 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:46.596798    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq" event={"ID":"c69432b1-9a81-4293-bd1c-4df1df813ba4","Type":"ContainerStarted","Data":"ff81bccf914454b0b3ebbc4c0b68984b915229268bc6a5b9fedcb69cdf5d52ce"}
Sep 17 18:29:46.596861 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:46.596847    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq" event={"ID":"c69432b1-9a81-4293-bd1c-4df1df813ba4","Type":"ContainerStarted","Data":"d1d25545d966d52a84fd328a3efbcc216a1d72746f8435e6bbfb324fd3781768"}
Sep 17 18:29:46.596923 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:46.596902    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq" event={"ID":"c69432b1-9a81-4293-bd1c-4df1df813ba4","Type":"ContainerStarted","Data":"b9cfe68729af2f650433b09e3e681816e46434a8a70d449597e9358f4f719966"}
Sep 17 18:29:46.596967 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:46.596950    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq" event={"ID":"c69432b1-9a81-4293-bd1c-4df1df813ba4","Type":"ContainerDied","Data":"70f5e3051ed3c6b7354ea274d2758fd5e61211cba003a43569ebc2e030dd6f73"}
Sep 17 18:29:46.597008 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:46.596996    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq" event={"ID":"c69432b1-9a81-4293-bd1c-4df1df813ba4","Type":"ContainerStarted","Data":"a39328ff6102138e9c8cf894bafb99610ad7381da6367424b81c8f0867138941"}
Sep 17 18:29:46.598659 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:46.598633    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5cb86" event={"ID":"947781e1-4423-435e-b44f-53ca352f8df7","Type":"ContainerStarted","Data":"4da309f147cb6588c00f8afd36da6dadfbbf2ef622377785fa37179ce9afb802"}
Sep 17 18:29:46.600557 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:46.600355    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-f96pg" event={"ID":"8462659d-7d75-4d56-b4c1-75dfc4aea577","Type":"ContainerStarted","Data":"27c904a73aafa0b7824af53d5d93aa26af8f263253f383369ebbc5c8db6cd716"}
Sep 17 18:29:46.602044 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:46.602024    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2t758" event={"ID":"10d63418-937e-4dee-a71f-592cfdff55ca","Type":"ContainerStarted","Data":"ddc235e6da8eb010707e50f9517e9962a5909498b076da8130b8d52948008cb3"}
Sep 17 18:29:46.603916 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:46.603895    2588 generic.go:356] "Generic (PLEG): container finished" podID="a7f6c88f-5861-4089-a4a8-75bb85c16c98" containerID="98be796bd37725816295db93357f5cb2679f9217ae4cdaacd1eb07b50088d25b" exitCode=0
Sep 17 18:29:46.604021 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:46.603962    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kpswk" event={"ID":"a7f6c88f-5861-4089-a4a8-75bb85c16c98","Type":"ContainerDied","Data":"98be796bd37725816295db93357f5cb2679f9217ae4cdaacd1eb07b50088d25b"}
Sep 17 18:29:46.606259 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:46.606232    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z97dh" event={"ID":"9254a7f2-3a79-421d-be0c-2cb1fbfe46fa","Type":"ContainerStarted","Data":"e5960e92a871e12126eec9e9096eec5a65e70fbbd6f025ac8da8a065889c0fc3"}
Sep 17 18:29:46.607865 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:46.607840    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-745b6" event={"ID":"61513726-d6fb-4e35-a39f-818bcb1a9596","Type":"ContainerStarted","Data":"1fa97aab957bb7a9d53d53704a3bcb868347c8d916876ee36715384557fb349f"}
Sep 17 18:29:46.616921 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:46.616875    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5cb86" podStartSLOduration=11.696543377 podStartE2EDuration="21.616861423s" totalImagesPullingTime="9.920318046s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:46.616652927 +0000 UTC m=+23.467658339" watchObservedRunningTime="2026-09-17 18:29:46.616861423 +0000 UTC m=+23.467866821"
Sep 17 18:29:46.617018 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:46.616960    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-132.ec2.internal" podStartSLOduration=19.616951078 podStartE2EDuration="19.616951078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:30.575194009 +0000 UTC m=+7.426199409" watchObservedRunningTime="2026-09-17 18:29:46.616951078 +0000 UTC m=+23.467956479"
Sep 17 18:29:46.653802 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:46.653734    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-745b6" podStartSLOduration=4.270394149 podStartE2EDuration="21.653712826s" totalImagesPullingTime="17.383318677s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:46.631345151 +0000 UTC m=+23.482350543" watchObservedRunningTime="2026-09-17 18:29:46.653712826 +0000 UTC m=+23.504718226"
Sep 17 18:29:46.670681 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:46.670624    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-z97dh" podStartSLOduration=4.259437656 podStartE2EDuration="21.670606436s" totalImagesPullingTime="17.41116878s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:46.67043731 +0000 UTC m=+23.521442771" watchObservedRunningTime="2026-09-17 18:29:46.670606436 +0000 UTC m=+23.521611905"
Sep 17 18:29:46.688367 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:46.688312    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-f96pg" podStartSLOduration=4.225962937 podStartE2EDuration="21.688292348s" totalImagesPullingTime="17.462329411s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:46.687479993 +0000 UTC m=+23.538485389" watchObservedRunningTime="2026-09-17 18:29:46.688292348 +0000 UTC m=+23.539297748"
Sep 17 18:29:46.702416 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:46.702366    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2t758" podStartSLOduration=8.520469974 podStartE2EDuration="21.702349168s" totalImagesPullingTime="13.181879194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:46.7022711 +0000 UTC m=+23.553276499" watchObservedRunningTime="2026-09-17 18:29:46.702349168 +0000 UTC m=+23.553354567"
Sep 17 18:29:46.797721 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:46.797475    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:46.877114 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:46.877085    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:46.966173 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:46.966152    2588 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 18:29:47.501380 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:47.501298    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bnq8f"
Sep 17 18:29:47.501380 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:47.501315    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:29:47.501642 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:47.501447    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bnq8f" podUID="1c01ccd8-9e9c-467f-a2af-50720f272038"
Sep 17 18:29:47.501690 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:47.501666    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxndc"
Sep 17 18:29:47.501845 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:47.501813    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs688" podUID="c554cd64-2695-47d3-999a-2d8bf8d7ef63"
Sep 17 18:29:47.501922 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:47.501900    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxndc" podUID="9530dc1d-51a9-4c20-aa0f-dbfbf912e02e"
Sep 17 18:29:47.526206 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:47.526117    2588 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T18:29:46.966168709Z","UUID":"8faf907a-f7fd-45ff-bcac-5b3a237069d0","Handler":null,"Name":"","Endpoint":""}
Sep 17 18:29:47.527888 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:47.527871    2588 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 18:29:47.528022 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:47.527902    2588 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 18:29:47.613517 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:47.613479    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kh6l" event={"ID":"f6048ce0-9af4-4205-baa1-c28be5018d95","Type":"ContainerStarted","Data":"53d7506375d08b04e5affabefd0657b8f8fc7e79fe88d7aaa5bf1e24c2cb349f"}
Sep 17 18:29:47.615477 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:47.615447    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qsksv" event={"ID":"7969c2b7-d790-4dbd-bb84-4886d90f38d7","Type":"ContainerStarted","Data":"3e9e336791530468148916f8606f824beaeb1fccb44829edb9482f0fd89b9da1"}
Sep 17 18:29:47.631049 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:47.631004    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qsksv" podStartSLOduration=5.218363688 podStartE2EDuration="22.630988749s" totalImagesPullingTime="17.412625061s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:47.630359799 +0000 UTC m=+24.481365199" watchObservedRunningTime="2026-09-17 18:29:47.630988749 +0000 UTC m=+24.481994153"
Sep 17 18:29:47.712019 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:47.711982    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2t758"
Sep 17 18:29:47.713195 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:47.713171    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2t758"
Sep 17 18:29:47.732267 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:47.732224    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:47.796964 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:47.796932    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:48.620363 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:48.620329    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kh6l" event={"ID":"f6048ce0-9af4-4205-baa1-c28be5018d95","Type":"ContainerStarted","Data":"4f613eeba1705ae416a3d808d653756c299335fa1c9648d3c5e9b5933b7cd484"}
Sep 17 18:29:48.620765 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:48.620639    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2t758"
Sep 17 18:29:48.621108 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:48.621088    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2t758"
Sep 17 18:29:48.638691 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:48.638640    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2kh6l" podStartSLOduration=4.263776806 podStartE2EDuration="23.638620265s" totalImagesPullingTime="19.374843459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:48.637077671 +0000 UTC m=+25.488083088" watchObservedRunningTime="2026-09-17 18:29:48.638620265 +0000 UTC m=+25.489625666"
Sep 17 18:29:48.707861 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:48.707832    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:49.500895 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:49.500859    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:29:49.501062 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:49.500859    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxndc"
Sep 17 18:29:49.501062 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:49.500986    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs688" podUID="c554cd64-2695-47d3-999a-2d8bf8d7ef63"
Sep 17 18:29:49.501177 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:49.501087    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxndc" podUID="9530dc1d-51a9-4c20-aa0f-dbfbf912e02e"
Sep 17 18:29:49.501177 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:49.500859    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bnq8f"
Sep 17 18:29:49.501283 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:49.501235    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bnq8f" podUID="1c01ccd8-9e9c-467f-a2af-50720f272038"
Sep 17 18:29:49.627309 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:49.627283    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovn-acl-logging/0.log"
Sep 17 18:29:49.627840 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:49.627802    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq" event={"ID":"c69432b1-9a81-4293-bd1c-4df1df813ba4","Type":"ContainerStarted","Data":"af0e8ab51d74b5fa7d868a32fdee2d8ef107109050c57444178b9e88dc61ead7"}
Sep 17 18:29:51.500681 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:51.500476    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:29:51.501323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:51.500476    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxndc"
Sep 17 18:29:51.501323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:51.500481    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bnq8f"
Sep 17 18:29:51.501323 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:51.500750    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs688" podUID="c554cd64-2695-47d3-999a-2d8bf8d7ef63"
Sep 17 18:29:51.501323 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:51.500881    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxndc" podUID="9530dc1d-51a9-4c20-aa0f-dbfbf912e02e"
Sep 17 18:29:51.501323 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:51.500994    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bnq8f" podUID="1c01ccd8-9e9c-467f-a2af-50720f272038"
Sep 17 18:29:51.633691 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:51.633658    2588 generic.go:356] "Generic (PLEG): container finished" podID="a7f6c88f-5861-4089-a4a8-75bb85c16c98" containerID="df5376aa65d4d40e530f8e8e7f8f93f9e801222f1831308275635b75d45f95f1" exitCode=0
Sep 17 18:29:51.633843 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:51.633739    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kpswk" event={"ID":"a7f6c88f-5861-4089-a4a8-75bb85c16c98","Type":"ContainerDied","Data":"df5376aa65d4d40e530f8e8e7f8f93f9e801222f1831308275635b75d45f95f1"}
Sep 17 18:29:51.637360 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:51.637341    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovn-acl-logging/0.log"
Sep 17 18:29:51.637796 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:51.637765    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq" event={"ID":"c69432b1-9a81-4293-bd1c-4df1df813ba4","Type":"ContainerStarted","Data":"8e837956b0621f9461a01b91a9bd20de1c9f69345c9b5e7120e7dd8a96f8063f"}
Sep 17 18:29:51.638036 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:51.638021    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:51.638165 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:51.638150    2588 scope.go:117] "RemoveContainer" containerID="70f5e3051ed3c6b7354ea274d2758fd5e61211cba003a43569ebc2e030dd6f73"
Sep 17 18:29:51.654479 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:51.654460    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:52.644762 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:52.644591    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovn-acl-logging/0.log"
Sep 17 18:29:52.645372 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:52.645092    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq" event={"ID":"c69432b1-9a81-4293-bd1c-4df1df813ba4","Type":"ContainerStarted","Data":"08706ce93977b0439977b608e8c17cab42f0cdd85edf1c82e4b3abca329e584e"}
Sep 17 18:29:52.645372 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:52.645240    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:52.645372 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:52.645293    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:52.661057 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:52.661036    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:29:52.680131 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:52.680083    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq" podStartSLOduration=10.251180147 podStartE2EDuration="27.680062347s" totalImagesPullingTime="17.4288822s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:52.678401318 +0000 UTC m=+29.529406744" watchObservedRunningTime="2026-09-17 18:29:52.680062347 +0000 UTC m=+29.531067747"
Sep 17 18:29:52.801787 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:52.801762    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fs688"]
Sep 17 18:29:52.801903 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:52.801891    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:29:52.802018 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:52.801980    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs688" podUID="c554cd64-2695-47d3-999a-2d8bf8d7ef63"
Sep 17 18:29:52.805801 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:52.805773    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bnq8f"]
Sep 17 18:29:52.805912 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:52.805893    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bnq8f"
Sep 17 18:29:52.806039 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:52.806014    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bnq8f" podUID="1c01ccd8-9e9c-467f-a2af-50720f272038"
Sep 17 18:29:52.806364 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:52.806345    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cxndc"]
Sep 17 18:29:52.806449 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:52.806436    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxndc"
Sep 17 18:29:52.806569 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:52.806541    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxndc" podUID="9530dc1d-51a9-4c20-aa0f-dbfbf912e02e"
Sep 17 18:29:53.649611 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:53.649579    2588 generic.go:356] "Generic (PLEG): container finished" podID="a7f6c88f-5861-4089-a4a8-75bb85c16c98" containerID="f6890753b690fb58de89ca313780d1d1ef0ae95c3fe55d8ecb7ca61b5c1f35bd" exitCode=0
Sep 17 18:29:53.650011 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:53.649673    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kpswk" event={"ID":"a7f6c88f-5861-4089-a4a8-75bb85c16c98","Type":"ContainerDied","Data":"f6890753b690fb58de89ca313780d1d1ef0ae95c3fe55d8ecb7ca61b5c1f35bd"}
Sep 17 18:29:54.501126 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:54.501095    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bnq8f"
Sep 17 18:29:54.501293 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:54.501203    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bnq8f" podUID="1c01ccd8-9e9c-467f-a2af-50720f272038"
Sep 17 18:29:54.501293 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:54.501095    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:29:54.501369 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:54.501112    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxndc"
Sep 17 18:29:54.501369 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:54.501315    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs688" podUID="c554cd64-2695-47d3-999a-2d8bf8d7ef63"
Sep 17 18:29:54.501441 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:54.501377    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxndc" podUID="9530dc1d-51a9-4c20-aa0f-dbfbf912e02e"
Sep 17 18:29:55.656954 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:55.656916    2588 generic.go:356] "Generic (PLEG): container finished" podID="a7f6c88f-5861-4089-a4a8-75bb85c16c98" containerID="2731033c9f61a917d3edd2799adce1e8d5f0c4d19405173e91d4e366d2e368cf" exitCode=0
Sep 17 18:29:55.657362 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:55.656970    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kpswk" event={"ID":"a7f6c88f-5861-4089-a4a8-75bb85c16c98","Type":"ContainerDied","Data":"2731033c9f61a917d3edd2799adce1e8d5f0c4d19405173e91d4e366d2e368cf"}
Sep 17 18:29:56.501197 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:56.501157    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:29:56.501393 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:56.501202    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bnq8f"
Sep 17 18:29:56.501393 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:56.501280    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxndc"
Sep 17 18:29:56.501393 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:56.501287    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs688" podUID="c554cd64-2695-47d3-999a-2d8bf8d7ef63"
Sep 17 18:29:56.501393 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:56.501344    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bnq8f" podUID="1c01ccd8-9e9c-467f-a2af-50720f272038"
Sep 17 18:29:56.501577 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:56.501451    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxndc" podUID="9530dc1d-51a9-4c20-aa0f-dbfbf912e02e"
Sep 17 18:29:57.233332 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:57.233296    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:57.300426 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:57.300393    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:57.372566 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:57.372344    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:58.292057 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:58.292017    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:58.360395 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:58.360358    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:58.501166 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:58.501136    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:29:58.501166 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:58.501156    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bnq8f"
Sep 17 18:29:58.501166 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:58.501159    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxndc"
Sep 17 18:29:58.501462 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:58.501254    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fs688" podUID="c554cd64-2695-47d3-999a-2d8bf8d7ef63"
Sep 17 18:29:58.501462 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:58.501330    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bnq8f" podUID="1c01ccd8-9e9c-467f-a2af-50720f272038"
Sep 17 18:29:58.501462 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:58.501413    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cxndc" podUID="9530dc1d-51a9-4c20-aa0f-dbfbf912e02e"
Sep 17 18:29:59.187037 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.186948    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c554cd64-2695-47d3-999a-2d8bf8d7ef63-original-pull-secret\") pod \"global-pull-secret-syncer-fs688\" (UID: \"c554cd64-2695-47d3-999a-2d8bf8d7ef63\") " pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:29:59.187037 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.186994    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c01ccd8-9e9c-467f-a2af-50720f272038-metrics-certs\") pod \"network-metrics-daemon-bnq8f\" (UID: \"1c01ccd8-9e9c-467f-a2af-50720f272038\") " pod="openshift-multus/network-metrics-daemon-bnq8f"
Sep 17 18:29:59.187315 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:59.187105    2588 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:59.187315 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:59.187113    2588 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:59.187315 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:59.187176    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c01ccd8-9e9c-467f-a2af-50720f272038-metrics-certs podName:1c01ccd8-9e9c-467f-a2af-50720f272038 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:31.187155939 +0000 UTC m=+68.038161341 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c01ccd8-9e9c-467f-a2af-50720f272038-metrics-certs") pod "network-metrics-daemon-bnq8f" (UID: "1c01ccd8-9e9c-467f-a2af-50720f272038") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:59.187315 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:59.187195    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c554cd64-2695-47d3-999a-2d8bf8d7ef63-original-pull-secret podName:c554cd64-2695-47d3-999a-2d8bf8d7ef63 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:31.187185465 +0000 UTC m=+68.038190848 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c554cd64-2695-47d3-999a-2d8bf8d7ef63-original-pull-secret") pod "global-pull-secret-syncer-fs688" (UID: "c554cd64-2695-47d3-999a-2d8bf8d7ef63") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:59.288251 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.288215    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcwlc\" (UniqueName: \"kubernetes.io/projected/9530dc1d-51a9-4c20-aa0f-dbfbf912e02e-kube-api-access-zcwlc\") pod \"network-check-target-cxndc\" (UID: \"9530dc1d-51a9-4c20-aa0f-dbfbf912e02e\") " pod="openshift-network-diagnostics/network-check-target-cxndc"
Sep 17 18:29:59.288442 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:59.288399    2588 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:59.288442 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:59.288426    2588 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:59.288442 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:59.288438    2588 projected.go:196] Error preparing data for projected volume kube-api-access-zcwlc for pod openshift-network-diagnostics/network-check-target-cxndc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:59.288610 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:59.288498    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9530dc1d-51a9-4c20-aa0f-dbfbf912e02e-kube-api-access-zcwlc podName:9530dc1d-51a9-4c20-aa0f-dbfbf912e02e nodeName:}" failed. No retries permitted until 2026-09-17 18:30:31.288480695 +0000 UTC m=+68.139486077 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zcwlc" (UniqueName: "kubernetes.io/projected/9530dc1d-51a9-4c20-aa0f-dbfbf912e02e-kube-api-access-zcwlc") pod "network-check-target-cxndc" (UID: "9530dc1d-51a9-4c20-aa0f-dbfbf912e02e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:59.517002 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.516924    2588 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-132.ec2.internal" event="NodeReady"
Sep 17 18:29:59.517475 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.517122    2588 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 18:29:59.581587 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.581554    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jq7kv"]
Sep 17 18:29:59.593443 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.593415    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6llb8"]
Sep 17 18:29:59.593619 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.593566    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jq7kv"
Sep 17 18:29:59.596250 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.596228    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 18:29:59.596380 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.596279    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 18:29:59.596380 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.596369    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cvvrd\""
Sep 17 18:29:59.605868 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.605833    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jq7kv"]
Sep 17 18:29:59.605868 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.605877    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6llb8"]
Sep 17 18:29:59.605868 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.605968    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6llb8"
Sep 17 18:29:59.608594 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.608572    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 18:29:59.608801 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.608612    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7c6q4\""
Sep 17 18:29:59.608801 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.608639    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 18:29:59.609558 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.609543    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 18:29:59.690772 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.690739    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a7075b4-9ee1-4495-86e5-b2377d9b1b06-tmp-dir\") pod \"dns-default-jq7kv\" (UID: \"6a7075b4-9ee1-4495-86e5-b2377d9b1b06\") " pod="openshift-dns/dns-default-jq7kv"
Sep 17 18:29:59.690772 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.690776    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/676d0454-70c3-4491-b9bc-7d934002b6be-cert\") pod \"ingress-canary-6llb8\" (UID: \"676d0454-70c3-4491-b9bc-7d934002b6be\") " pod="openshift-ingress-canary/ingress-canary-6llb8"
Sep 17 18:29:59.690995 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.690823    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a7075b4-9ee1-4495-86e5-b2377d9b1b06-config-volume\") pod \"dns-default-jq7kv\" (UID: \"6a7075b4-9ee1-4495-86e5-b2377d9b1b06\") " pod="openshift-dns/dns-default-jq7kv"
Sep 17 18:29:59.690995 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.690860    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a7075b4-9ee1-4495-86e5-b2377d9b1b06-metrics-tls\") pod \"dns-default-jq7kv\" (UID: \"6a7075b4-9ee1-4495-86e5-b2377d9b1b06\") " pod="openshift-dns/dns-default-jq7kv"
Sep 17 18:29:59.690995 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.690899    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9qmc\" (UniqueName: \"kubernetes.io/projected/676d0454-70c3-4491-b9bc-7d934002b6be-kube-api-access-b9qmc\") pod \"ingress-canary-6llb8\" (UID: \"676d0454-70c3-4491-b9bc-7d934002b6be\") " pod="openshift-ingress-canary/ingress-canary-6llb8"
Sep 17 18:29:59.690995 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.690914    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khrbh\" (UniqueName: \"kubernetes.io/projected/6a7075b4-9ee1-4495-86e5-b2377d9b1b06-kube-api-access-khrbh\") pod \"dns-default-jq7kv\" (UID: \"6a7075b4-9ee1-4495-86e5-b2377d9b1b06\") " pod="openshift-dns/dns-default-jq7kv"
Sep 17 18:29:59.792086 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.792006    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/676d0454-70c3-4491-b9bc-7d934002b6be-cert\") pod \"ingress-canary-6llb8\" (UID: \"676d0454-70c3-4491-b9bc-7d934002b6be\") " pod="openshift-ingress-canary/ingress-canary-6llb8"
Sep 17 18:29:59.792086 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.792070    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a7075b4-9ee1-4495-86e5-b2377d9b1b06-config-volume\") pod \"dns-default-jq7kv\" (UID: \"6a7075b4-9ee1-4495-86e5-b2377d9b1b06\") " pod="openshift-dns/dns-default-jq7kv"
Sep 17 18:29:59.792323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.792102    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a7075b4-9ee1-4495-86e5-b2377d9b1b06-metrics-tls\") pod \"dns-default-jq7kv\" (UID: \"6a7075b4-9ee1-4495-86e5-b2377d9b1b06\") " pod="openshift-dns/dns-default-jq7kv"
Sep 17 18:29:59.792323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.792152    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9qmc\" (UniqueName: \"kubernetes.io/projected/676d0454-70c3-4491-b9bc-7d934002b6be-kube-api-access-b9qmc\") pod \"ingress-canary-6llb8\" (UID: \"676d0454-70c3-4491-b9bc-7d934002b6be\") " pod="openshift-ingress-canary/ingress-canary-6llb8"
Sep 17 18:29:59.792323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.792175    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khrbh\" (UniqueName: \"kubernetes.io/projected/6a7075b4-9ee1-4495-86e5-b2377d9b1b06-kube-api-access-khrbh\") pod \"dns-default-jq7kv\" (UID: \"6a7075b4-9ee1-4495-86e5-b2377d9b1b06\") " pod="openshift-dns/dns-default-jq7kv"
Sep 17 18:29:59.792323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.792221    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a7075b4-9ee1-4495-86e5-b2377d9b1b06-tmp-dir\") pod \"dns-default-jq7kv\" (UID: \"6a7075b4-9ee1-4495-86e5-b2377d9b1b06\") " pod="openshift-dns/dns-default-jq7kv"
Sep 17 18:29:59.792323 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:59.792225    2588 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:29:59.792595 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:29:59.792338    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a7075b4-9ee1-4495-86e5-b2377d9b1b06-metrics-tls podName:6a7075b4-9ee1-4495-86e5-b2377d9b1b06 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:00.292316049 +0000 UTC m=+37.143321441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a7075b4-9ee1-4495-86e5-b2377d9b1b06-metrics-tls") pod "dns-default-jq7kv" (UID: "6a7075b4-9ee1-4495-86e5-b2377d9b1b06") : secret "dns-default-metrics-tls" not found
Sep 17 18:29:59.792595 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.792556    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a7075b4-9ee1-4495-86e5-b2377d9b1b06-tmp-dir\") pod \"dns-default-jq7kv\" (UID: \"6a7075b4-9ee1-4495-86e5-b2377d9b1b06\") " pod="openshift-dns/dns-default-jq7kv"
Sep 17 18:29:59.792691 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.792676    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a7075b4-9ee1-4495-86e5-b2377d9b1b06-config-volume\") pod \"dns-default-jq7kv\" (UID: \"6a7075b4-9ee1-4495-86e5-b2377d9b1b06\") " pod="openshift-dns/dns-default-jq7kv"
Sep 17 18:29:59.796496 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.796380    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/676d0454-70c3-4491-b9bc-7d934002b6be-cert\") pod \"ingress-canary-6llb8\" (UID: \"676d0454-70c3-4491-b9bc-7d934002b6be\") " pod="openshift-ingress-canary/ingress-canary-6llb8"
Sep 17 18:29:59.804379 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.804352    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khrbh\" (UniqueName: \"kubernetes.io/projected/6a7075b4-9ee1-4495-86e5-b2377d9b1b06-kube-api-access-khrbh\") pod \"dns-default-jq7kv\" (UID: \"6a7075b4-9ee1-4495-86e5-b2377d9b1b06\") " pod="openshift-dns/dns-default-jq7kv"
Sep 17 18:29:59.804502 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.804436    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9qmc\" (UniqueName: \"kubernetes.io/projected/676d0454-70c3-4491-b9bc-7d934002b6be-kube-api-access-b9qmc\") pod \"ingress-canary-6llb8\" (UID: \"676d0454-70c3-4491-b9bc-7d934002b6be\") " pod="openshift-ingress-canary/ingress-canary-6llb8"
Sep 17 18:29:59.913628 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:29:59.913590    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6llb8"
Sep 17 18:30:00.234805 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:00.234769    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6llb8"]
Sep 17 18:30:00.238062 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:30:00.238036    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod676d0454_70c3_4491_b9bc_7d934002b6be.slice/crio-88788758a307bdb87bfe8b64a8fe8b606340cc00a9de242b1f8a3bf71d8a8f60 WatchSource:0}: Error finding container 88788758a307bdb87bfe8b64a8fe8b606340cc00a9de242b1f8a3bf71d8a8f60: Status 404 returned error can't find the container with id 88788758a307bdb87bfe8b64a8fe8b606340cc00a9de242b1f8a3bf71d8a8f60
Sep 17 18:30:00.296596 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:00.296536    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a7075b4-9ee1-4495-86e5-b2377d9b1b06-metrics-tls\") pod \"dns-default-jq7kv\" (UID: \"6a7075b4-9ee1-4495-86e5-b2377d9b1b06\") " pod="openshift-dns/dns-default-jq7kv"
Sep 17 18:30:00.296713 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:00.296648    2588 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:30:00.296753 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:00.296712    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a7075b4-9ee1-4495-86e5-b2377d9b1b06-metrics-tls podName:6a7075b4-9ee1-4495-86e5-b2377d9b1b06 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:01.296696205 +0000 UTC m=+38.147701581 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a7075b4-9ee1-4495-86e5-b2377d9b1b06-metrics-tls") pod "dns-default-jq7kv" (UID: "6a7075b4-9ee1-4495-86e5-b2377d9b1b06") : secret "dns-default-metrics-tls" not found
Sep 17 18:30:00.500521 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:00.500480    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:30:00.500711 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:00.500493    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bnq8f"
Sep 17 18:30:00.500711 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:00.500506    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxndc"
Sep 17 18:30:00.503401 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:00.503379    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 18:30:00.503522 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:00.503415    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 18:30:00.503522 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:00.503464    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 18:30:00.503522 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:00.503509    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-twvp2\""
Sep 17 18:30:00.503706 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:00.503683    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 18:30:00.503760 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:00.503724    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-58cdj\""
Sep 17 18:30:00.672051 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:00.671961    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6llb8" event={"ID":"676d0454-70c3-4491-b9bc-7d934002b6be","Type":"ContainerStarted","Data":"88788758a307bdb87bfe8b64a8fe8b606340cc00a9de242b1f8a3bf71d8a8f60"}
Sep 17 18:30:00.675037 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:00.675009    2588 generic.go:356] "Generic (PLEG): container finished" podID="a7f6c88f-5861-4089-a4a8-75bb85c16c98" containerID="f52791d71f089b2a7f60f74c4372f1ad074fad8dd56782bbc48446d9385521f5" exitCode=0
Sep 17 18:30:00.675148 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:00.675069    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kpswk" event={"ID":"a7f6c88f-5861-4089-a4a8-75bb85c16c98","Type":"ContainerDied","Data":"f52791d71f089b2a7f60f74c4372f1ad074fad8dd56782bbc48446d9385521f5"}
Sep 17 18:30:01.307066 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:01.307023    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a7075b4-9ee1-4495-86e5-b2377d9b1b06-metrics-tls\") pod \"dns-default-jq7kv\" (UID: \"6a7075b4-9ee1-4495-86e5-b2377d9b1b06\") " pod="openshift-dns/dns-default-jq7kv"
Sep 17 18:30:01.307324 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:01.307190    2588 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:30:01.307324 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:01.307256    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a7075b4-9ee1-4495-86e5-b2377d9b1b06-metrics-tls podName:6a7075b4-9ee1-4495-86e5-b2377d9b1b06 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:03.30723857 +0000 UTC m=+40.158243947 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a7075b4-9ee1-4495-86e5-b2377d9b1b06-metrics-tls") pod "dns-default-jq7kv" (UID: "6a7075b4-9ee1-4495-86e5-b2377d9b1b06") : secret "dns-default-metrics-tls" not found
Sep 17 18:30:01.681895 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:01.681816    2588 generic.go:356] "Generic (PLEG): container finished" podID="a7f6c88f-5861-4089-a4a8-75bb85c16c98" containerID="c745289e4bd47991c9f7cfd85b31da73183fb6828de0c74eb4f82220b3a5ed22" exitCode=0
Sep 17 18:30:01.681895 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:01.681867    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kpswk" event={"ID":"a7f6c88f-5861-4089-a4a8-75bb85c16c98","Type":"ContainerDied","Data":"c745289e4bd47991c9f7cfd85b31da73183fb6828de0c74eb4f82220b3a5ed22"}
Sep 17 18:30:01.925589 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:01.925367    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-x69bl"]
Sep 17 18:30:01.945745 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:01.945306    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-j5s56"]
Sep 17 18:30:01.946886 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:01.946028    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-x69bl"
Sep 17 18:30:01.948597 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:01.948569    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:30:01.948720 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:01.948596    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 17 18:30:01.948898 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:01.948878    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-l5wxx\""
Sep 17 18:30:01.954382 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:01.954359    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-x69bl"]
Sep 17 18:30:01.954510 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:01.954401    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-xlklb"]
Sep 17 18:30:01.954593 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:01.954516    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-j5s56"
Sep 17 18:30:01.958407 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:01.958387    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 17 18:30:01.958514 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:01.958390    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 18:30:01.959854 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:01.959821    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 18:30:01.959973 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:01.959888    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-zcf4g\""
Sep 17 18:30:01.960104 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:01.960018    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 17 18:30:01.962482 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:01.962461    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-j5s56"]
Sep 17 18:30:01.962665 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:01.962596    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-xlklb"
Sep 17 18:30:01.965286 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:01.965266    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:30:01.966461 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:01.966314    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 17 18:30:01.970336 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:01.970315    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-xlklb"]
Sep 17 18:30:01.970933 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:01.970871    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-gbmgt\""
Sep 17 18:30:01.973561 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:01.973509    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 17 18:30:01.974151 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:01.973942    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 17 18:30:01.982183 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:01.982166    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 17 18:30:02.012223 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.012188    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9htnn\" (UniqueName: \"kubernetes.io/projected/96541b99-da29-416c-916f-91fbb7207ef1-kube-api-access-9htnn\") pod \"volume-data-source-validator-5c4db8bdf9-x69bl\" (UID: \"96541b99-da29-416c-916f-91fbb7207ef1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-x69bl"
Sep 17 18:30:02.019101 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.019073    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-vsp4l"]
Sep 17 18:30:02.036079 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.036047    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-cf8bw"]
Sep 17 18:30:02.036237 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.036196    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-vsp4l"
Sep 17 18:30:02.040578 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.040557    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-qp8s8\""
Sep 17 18:30:02.055038 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.055014    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8xqn"]
Sep 17 18:30:02.055164 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.055121    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-cf8bw"
Sep 17 18:30:02.058815 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.058708    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 17 18:30:02.058815 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.058720    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 18:30:02.058815 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.058765    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-5ztrn\""
Sep 17 18:30:02.058815 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.058765    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 17 18:30:02.059333 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.059312    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 18:30:02.064033 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.064014    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 17 18:30:02.075441 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.075410    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-vsp4l"]
Sep 17 18:30:02.075441 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.075444    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-cf8bw"]
Sep 17 18:30:02.075608 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.075468    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8xqn"]
Sep 17 18:30:02.075608 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.075521    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8xqn"
Sep 17 18:30:02.078385 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.078117    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:30:02.078385 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.078134    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-q5k2h\""
Sep 17 18:30:02.078385 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.078160    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 17 18:30:02.078385 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.078186    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 17 18:30:02.113330 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.113305    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/49a517a5-36f1-4482-b8fc-fa78fb902040-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-j5s56\" (UID: \"49a517a5-36f1-4482-b8fc-fa78fb902040\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-j5s56"
Sep 17 18:30:02.113488 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.113342    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx4xp\" (UniqueName: \"kubernetes.io/projected/fd80d0a0-6acf-4632-8484-07395506d4c0-kube-api-access-hx4xp\") pod \"network-check-source-7cbc45fddc-vsp4l\" (UID: \"fd80d0a0-6acf-4632-8484-07395506d4c0\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-vsp4l"
Sep 17 18:30:02.113488 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.113396    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/07ee31e0-43bb-4940-837d-969bd840492b-config\") pod \"console-operator-fd4bb46c-xlklb\" (UID: \"07ee31e0-43bb-4940-837d-969bd840492b\") " pod="openshift-console-operator/console-operator-fd4bb46c-xlklb"
Sep 17 18:30:02.113488 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.113454    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/49a517a5-36f1-4482-b8fc-fa78fb902040-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-j5s56\" (UID: \"49a517a5-36f1-4482-b8fc-fa78fb902040\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-j5s56"
Sep 17 18:30:02.113680 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.113501    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9htnn\" (UniqueName: \"kubernetes.io/projected/96541b99-da29-416c-916f-91fbb7207ef1-kube-api-access-9htnn\") pod \"volume-data-source-validator-5c4db8bdf9-x69bl\" (UID: \"96541b99-da29-416c-916f-91fbb7207ef1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-x69bl"
Sep 17 18:30:02.113680 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.113548    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46vxz\" (UniqueName: \"kubernetes.io/projected/49a517a5-36f1-4482-b8fc-fa78fb902040-kube-api-access-46vxz\") pod \"cluster-monitoring-operator-847b8bd8d7-j5s56\" (UID: \"49a517a5-36f1-4482-b8fc-fa78fb902040\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-j5s56"
Sep 17 18:30:02.113680 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.113573    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/07ee31e0-43bb-4940-837d-969bd840492b-serving-cert\") pod \"console-operator-fd4bb46c-xlklb\" (UID: \"07ee31e0-43bb-4940-837d-969bd840492b\") " pod="openshift-console-operator/console-operator-fd4bb46c-xlklb"
Sep 17 18:30:02.113680 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.113606    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/07ee31e0-43bb-4940-837d-969bd840492b-trusted-ca\") pod \"console-operator-fd4bb46c-xlklb\" (UID: \"07ee31e0-43bb-4940-837d-969bd840492b\") " pod="openshift-console-operator/console-operator-fd4bb46c-xlklb"
Sep 17 18:30:02.113680 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.113657    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7bgn\" (UniqueName: \"kubernetes.io/projected/07ee31e0-43bb-4940-837d-969bd840492b-kube-api-access-k7bgn\") pod \"console-operator-fd4bb46c-xlklb\" (UID: \"07ee31e0-43bb-4940-837d-969bd840492b\") " pod="openshift-console-operator/console-operator-fd4bb46c-xlklb"
Sep 17 18:30:02.122478 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.122441    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9htnn\" (UniqueName: \"kubernetes.io/projected/96541b99-da29-416c-916f-91fbb7207ef1-kube-api-access-9htnn\") pod \"volume-data-source-validator-5c4db8bdf9-x69bl\" (UID: \"96541b99-da29-416c-916f-91fbb7207ef1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-x69bl"
Sep 17 18:30:02.161834 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.161803    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-587f5c755b-xds6m"]
Sep 17 18:30:02.184514 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.184482    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:02.187084 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.187058    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-587f5c755b-xds6m"]
Sep 17 18:30:02.187209 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.187167    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:02.190989 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.190965    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 18:30:02.190989 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.190983    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 18:30:02.191168 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.191094    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 18:30:02.191390 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.191372    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pndxg\""
Sep 17 18:30:02.197100 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.197076    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 18:30:02.214642 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.214618    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63716ecc-bcb1-44d0-96c4-40ac49599a79-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-cf8bw\" (UID: \"63716ecc-bcb1-44d0-96c4-40ac49599a79\") " pod="openshift-insights/insights-operator-6f9c644dd9-cf8bw"
Sep 17 18:30:02.214759 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.214655    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46vxz\" (UniqueName: \"kubernetes.io/projected/49a517a5-36f1-4482-b8fc-fa78fb902040-kube-api-access-46vxz\") pod \"cluster-monitoring-operator-847b8bd8d7-j5s56\" (UID: \"49a517a5-36f1-4482-b8fc-fa78fb902040\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-j5s56"
Sep 17 18:30:02.214759 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.214682    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/07ee31e0-43bb-4940-837d-969bd840492b-serving-cert\") pod \"console-operator-fd4bb46c-xlklb\" (UID: \"07ee31e0-43bb-4940-837d-969bd840492b\") " pod="openshift-console-operator/console-operator-fd4bb46c-xlklb"
Sep 17 18:30:02.214759 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.214746    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/07ee31e0-43bb-4940-837d-969bd840492b-trusted-ca\") pod \"console-operator-fd4bb46c-xlklb\" (UID: \"07ee31e0-43bb-4940-837d-969bd840492b\") " pod="openshift-console-operator/console-operator-fd4bb46c-xlklb"
Sep 17 18:30:02.214907 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.214784    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7bgn\" (UniqueName: \"kubernetes.io/projected/07ee31e0-43bb-4940-837d-969bd840492b-kube-api-access-k7bgn\") pod \"console-operator-fd4bb46c-xlklb\" (UID: \"07ee31e0-43bb-4940-837d-969bd840492b\") " pod="openshift-console-operator/console-operator-fd4bb46c-xlklb"
Sep 17 18:30:02.214907 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.214818    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/63716ecc-bcb1-44d0-96c4-40ac49599a79-snapshots\") pod \"insights-operator-6f9c644dd9-cf8bw\" (UID: \"63716ecc-bcb1-44d0-96c4-40ac49599a79\") " pod="openshift-insights/insights-operator-6f9c644dd9-cf8bw"
Sep 17 18:30:02.214907 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.214848    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63716ecc-bcb1-44d0-96c4-40ac49599a79-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-cf8bw\" (UID: \"63716ecc-bcb1-44d0-96c4-40ac49599a79\") " pod="openshift-insights/insights-operator-6f9c644dd9-cf8bw"
Sep 17 18:30:02.214907 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.214881    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/49a517a5-36f1-4482-b8fc-fa78fb902040-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-j5s56\" (UID: \"49a517a5-36f1-4482-b8fc-fa78fb902040\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-j5s56"
Sep 17 18:30:02.215085 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.214911    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx4xp\" (UniqueName: \"kubernetes.io/projected/fd80d0a0-6acf-4632-8484-07395506d4c0-kube-api-access-hx4xp\") pod \"network-check-source-7cbc45fddc-vsp4l\" (UID: \"fd80d0a0-6acf-4632-8484-07395506d4c0\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-vsp4l"
Sep 17 18:30:02.215085 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.214945    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d60900b-84cb-40d3-9ece-74f02821725d-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-p8xqn\" (UID: \"9d60900b-84cb-40d3-9ece-74f02821725d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8xqn"
Sep 17 18:30:02.215085 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.214974    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7pzq\" (UniqueName: \"kubernetes.io/projected/9d60900b-84cb-40d3-9ece-74f02821725d-kube-api-access-d7pzq\") pod \"cluster-samples-operator-685d7d68b5-p8xqn\" (UID: \"9d60900b-84cb-40d3-9ece-74f02821725d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8xqn"
Sep 17 18:30:02.215085 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.215019    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/07ee31e0-43bb-4940-837d-969bd840492b-config\") pod \"console-operator-fd4bb46c-xlklb\" (UID: \"07ee31e0-43bb-4940-837d-969bd840492b\") " pod="openshift-console-operator/console-operator-fd4bb46c-xlklb"
Sep 17 18:30:02.215085 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.215044    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/63716ecc-bcb1-44d0-96c4-40ac49599a79-serving-cert\") pod \"insights-operator-6f9c644dd9-cf8bw\" (UID: \"63716ecc-bcb1-44d0-96c4-40ac49599a79\") " pod="openshift-insights/insights-operator-6f9c644dd9-cf8bw"
Sep 17 18:30:02.215279 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.215100    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/49a517a5-36f1-4482-b8fc-fa78fb902040-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-j5s56\" (UID: \"49a517a5-36f1-4482-b8fc-fa78fb902040\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-j5s56"
Sep 17 18:30:02.215279 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.215133    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63716ecc-bcb1-44d0-96c4-40ac49599a79-tmp\") pod \"insights-operator-6f9c644dd9-cf8bw\" (UID: \"63716ecc-bcb1-44d0-96c4-40ac49599a79\") " pod="openshift-insights/insights-operator-6f9c644dd9-cf8bw"
Sep 17 18:30:02.215279 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.215156    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjqhw\" (UniqueName: \"kubernetes.io/projected/63716ecc-bcb1-44d0-96c4-40ac49599a79-kube-api-access-qjqhw\") pod \"insights-operator-6f9c644dd9-cf8bw\" (UID: \"63716ecc-bcb1-44d0-96c4-40ac49599a79\") " pod="openshift-insights/insights-operator-6f9c644dd9-cf8bw"
Sep 17 18:30:02.215279 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:02.215190    2588 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 18:30:02.215279 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:02.215256    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49a517a5-36f1-4482-b8fc-fa78fb902040-cluster-monitoring-operator-tls podName:49a517a5-36f1-4482-b8fc-fa78fb902040 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:02.715237246 +0000 UTC m=+39.566242636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/49a517a5-36f1-4482-b8fc-fa78fb902040-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-j5s56" (UID: "49a517a5-36f1-4482-b8fc-fa78fb902040") : secret "cluster-monitoring-operator-tls" not found
Sep 17 18:30:02.215884 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.215867    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/07ee31e0-43bb-4940-837d-969bd840492b-config\") pod \"console-operator-fd4bb46c-xlklb\" (UID: \"07ee31e0-43bb-4940-837d-969bd840492b\") " pod="openshift-console-operator/console-operator-fd4bb46c-xlklb"
Sep 17 18:30:02.215974 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.215901    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/07ee31e0-43bb-4940-837d-969bd840492b-trusted-ca\") pod \"console-operator-fd4bb46c-xlklb\" (UID: \"07ee31e0-43bb-4940-837d-969bd840492b\") " pod="openshift-console-operator/console-operator-fd4bb46c-xlklb"
Sep 17 18:30:02.216066 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.216045    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/49a517a5-36f1-4482-b8fc-fa78fb902040-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-j5s56\" (UID: \"49a517a5-36f1-4482-b8fc-fa78fb902040\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-j5s56"
Sep 17 18:30:02.217371 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.217353    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/07ee31e0-43bb-4940-837d-969bd840492b-serving-cert\") pod \"console-operator-fd4bb46c-xlklb\" (UID: \"07ee31e0-43bb-4940-837d-969bd840492b\") " pod="openshift-console-operator/console-operator-fd4bb46c-xlklb"
Sep 17 18:30:02.224260 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.224229    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx4xp\" (UniqueName: \"kubernetes.io/projected/fd80d0a0-6acf-4632-8484-07395506d4c0-kube-api-access-hx4xp\") pod \"network-check-source-7cbc45fddc-vsp4l\" (UID: \"fd80d0a0-6acf-4632-8484-07395506d4c0\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-vsp4l"
Sep 17 18:30:02.224738 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.224717    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46vxz\" (UniqueName: \"kubernetes.io/projected/49a517a5-36f1-4482-b8fc-fa78fb902040-kube-api-access-46vxz\") pod \"cluster-monitoring-operator-847b8bd8d7-j5s56\" (UID: \"49a517a5-36f1-4482-b8fc-fa78fb902040\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-j5s56"
Sep 17 18:30:02.225009 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.224986    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7bgn\" (UniqueName: \"kubernetes.io/projected/07ee31e0-43bb-4940-837d-969bd840492b-kube-api-access-k7bgn\") pod \"console-operator-fd4bb46c-xlklb\" (UID: \"07ee31e0-43bb-4940-837d-969bd840492b\") " pod="openshift-console-operator/console-operator-fd4bb46c-xlklb"
Sep 17 18:30:02.258575 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.258543    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-x69bl"
Sep 17 18:30:02.273960 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.273929    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:02.281904 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.281878    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-xlklb"
Sep 17 18:30:02.315907 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.315879    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63716ecc-bcb1-44d0-96c4-40ac49599a79-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-cf8bw\" (UID: \"63716ecc-bcb1-44d0-96c4-40ac49599a79\") " pod="openshift-insights/insights-operator-6f9c644dd9-cf8bw"
Sep 17 18:30:02.316035 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.315930    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c6567c1f-021d-488b-8e11-8890cd9a8152-image-registry-private-configuration\") pod \"image-registry-587f5c755b-xds6m\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") " pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:02.316035 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.315962    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c6567c1f-021d-488b-8e11-8890cd9a8152-registry-certificates\") pod \"image-registry-587f5c755b-xds6m\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") " pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:02.316035 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.316002    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d60900b-84cb-40d3-9ece-74f02821725d-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-p8xqn\" (UID: \"9d60900b-84cb-40d3-9ece-74f02821725d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8xqn"
Sep 17 18:30:02.316035 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.316029    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6567c1f-021d-488b-8e11-8890cd9a8152-bound-sa-token\") pod \"image-registry-587f5c755b-xds6m\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") " pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:02.316236 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:02.316161    2588 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 18:30:02.316236 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.316162    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6567c1f-021d-488b-8e11-8890cd9a8152-registry-tls\") pod \"image-registry-587f5c755b-xds6m\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") " pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:02.316236 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:02.316223    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d60900b-84cb-40d3-9ece-74f02821725d-samples-operator-tls podName:9d60900b-84cb-40d3-9ece-74f02821725d nodeName:}" failed. No retries permitted until 2026-09-17 18:30:02.816208677 +0000 UTC m=+39.667214053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9d60900b-84cb-40d3-9ece-74f02821725d-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-p8xqn" (UID: "9d60900b-84cb-40d3-9ece-74f02821725d") : secret "samples-operator-tls" not found
Sep 17 18:30:02.316380 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.316294    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c6567c1f-021d-488b-8e11-8890cd9a8152-ca-trust-extracted\") pod \"image-registry-587f5c755b-xds6m\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") " pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:02.316380 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.316328    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/63716ecc-bcb1-44d0-96c4-40ac49599a79-snapshots\") pod \"insights-operator-6f9c644dd9-cf8bw\" (UID: \"63716ecc-bcb1-44d0-96c4-40ac49599a79\") " pod="openshift-insights/insights-operator-6f9c644dd9-cf8bw"
Sep 17 18:30:02.316380 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.316359    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c6567c1f-021d-488b-8e11-8890cd9a8152-installation-pull-secrets\") pod \"image-registry-587f5c755b-xds6m\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") " pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:02.316510 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.316389    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlhh7\" (UniqueName: \"kubernetes.io/projected/c6567c1f-021d-488b-8e11-8890cd9a8152-kube-api-access-qlhh7\") pod \"image-registry-587f5c755b-xds6m\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") " pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:02.316510 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.316494    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63716ecc-bcb1-44d0-96c4-40ac49599a79-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-cf8bw\" (UID: \"63716ecc-bcb1-44d0-96c4-40ac49599a79\") " pod="openshift-insights/insights-operator-6f9c644dd9-cf8bw"
Sep 17 18:30:02.316633 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.316501    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7pzq\" (UniqueName: \"kubernetes.io/projected/9d60900b-84cb-40d3-9ece-74f02821725d-kube-api-access-d7pzq\") pod \"cluster-samples-operator-685d7d68b5-p8xqn\" (UID: \"9d60900b-84cb-40d3-9ece-74f02821725d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8xqn"
Sep 17 18:30:02.316685 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.316643    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/63716ecc-bcb1-44d0-96c4-40ac49599a79-serving-cert\") pod \"insights-operator-6f9c644dd9-cf8bw\" (UID: \"63716ecc-bcb1-44d0-96c4-40ac49599a79\") " pod="openshift-insights/insights-operator-6f9c644dd9-cf8bw"
Sep 17 18:30:02.316685 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.316674    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63716ecc-bcb1-44d0-96c4-40ac49599a79-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-cf8bw\" (UID: \"63716ecc-bcb1-44d0-96c4-40ac49599a79\") " pod="openshift-insights/insights-operator-6f9c644dd9-cf8bw"
Sep 17 18:30:02.316785 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.316713    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63716ecc-bcb1-44d0-96c4-40ac49599a79-tmp\") pod \"insights-operator-6f9c644dd9-cf8bw\" (UID: \"63716ecc-bcb1-44d0-96c4-40ac49599a79\") " pod="openshift-insights/insights-operator-6f9c644dd9-cf8bw"
Sep 17 18:30:02.316785 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.316748    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjqhw\" (UniqueName: \"kubernetes.io/projected/63716ecc-bcb1-44d0-96c4-40ac49599a79-kube-api-access-qjqhw\") pod \"insights-operator-6f9c644dd9-cf8bw\" (UID: \"63716ecc-bcb1-44d0-96c4-40ac49599a79\") " pod="openshift-insights/insights-operator-6f9c644dd9-cf8bw"
Sep 17 18:30:02.316785 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.316776    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6567c1f-021d-488b-8e11-8890cd9a8152-trusted-ca\") pod \"image-registry-587f5c755b-xds6m\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") " pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:02.317017 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.316996    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/63716ecc-bcb1-44d0-96c4-40ac49599a79-snapshots\") pod \"insights-operator-6f9c644dd9-cf8bw\" (UID: \"63716ecc-bcb1-44d0-96c4-40ac49599a79\") " pod="openshift-insights/insights-operator-6f9c644dd9-cf8bw"
Sep 17 18:30:02.317143 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.317118    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63716ecc-bcb1-44d0-96c4-40ac49599a79-tmp\") pod \"insights-operator-6f9c644dd9-cf8bw\" (UID: \"63716ecc-bcb1-44d0-96c4-40ac49599a79\") " pod="openshift-insights/insights-operator-6f9c644dd9-cf8bw"
Sep 17 18:30:02.317473 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.317450    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63716ecc-bcb1-44d0-96c4-40ac49599a79-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-cf8bw\" (UID: \"63716ecc-bcb1-44d0-96c4-40ac49599a79\") " pod="openshift-insights/insights-operator-6f9c644dd9-cf8bw"
Sep 17 18:30:02.319543 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.319513    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/63716ecc-bcb1-44d0-96c4-40ac49599a79-serving-cert\") pod \"insights-operator-6f9c644dd9-cf8bw\" (UID: \"63716ecc-bcb1-44d0-96c4-40ac49599a79\") " pod="openshift-insights/insights-operator-6f9c644dd9-cf8bw"
Sep 17 18:30:02.324895 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.324861    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjqhw\" (UniqueName: \"kubernetes.io/projected/63716ecc-bcb1-44d0-96c4-40ac49599a79-kube-api-access-qjqhw\") pod \"insights-operator-6f9c644dd9-cf8bw\" (UID: \"63716ecc-bcb1-44d0-96c4-40ac49599a79\") " pod="openshift-insights/insights-operator-6f9c644dd9-cf8bw"
Sep 17 18:30:02.325013 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.324978    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7pzq\" (UniqueName: \"kubernetes.io/projected/9d60900b-84cb-40d3-9ece-74f02821725d-kube-api-access-d7pzq\") pod \"cluster-samples-operator-685d7d68b5-p8xqn\" (UID: \"9d60900b-84cb-40d3-9ece-74f02821725d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8xqn"
Sep 17 18:30:02.347626 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.347604    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-vsp4l"
Sep 17 18:30:02.365517 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.365494    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-cf8bw"
Sep 17 18:30:02.417909 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.417864    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6567c1f-021d-488b-8e11-8890cd9a8152-registry-tls\") pod \"image-registry-587f5c755b-xds6m\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") " pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:02.418062 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.417931    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c6567c1f-021d-488b-8e11-8890cd9a8152-ca-trust-extracted\") pod \"image-registry-587f5c755b-xds6m\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") " pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:02.418062 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.417960    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c6567c1f-021d-488b-8e11-8890cd9a8152-installation-pull-secrets\") pod \"image-registry-587f5c755b-xds6m\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") " pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:02.418062 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.417981    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlhh7\" (UniqueName: \"kubernetes.io/projected/c6567c1f-021d-488b-8e11-8890cd9a8152-kube-api-access-qlhh7\") pod \"image-registry-587f5c755b-xds6m\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") " pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:02.418062 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.418027    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6567c1f-021d-488b-8e11-8890cd9a8152-trusted-ca\") pod \"image-registry-587f5c755b-xds6m\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") " pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:02.418062 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:02.418045    2588 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:30:02.418062 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.418057    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c6567c1f-021d-488b-8e11-8890cd9a8152-image-registry-private-configuration\") pod \"image-registry-587f5c755b-xds6m\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") " pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:02.418318 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:02.418068    2588 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-587f5c755b-xds6m: secret "image-registry-tls" not found
Sep 17 18:30:02.418318 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.418089    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c6567c1f-021d-488b-8e11-8890cd9a8152-registry-certificates\") pod \"image-registry-587f5c755b-xds6m\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") " pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:02.418318 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:02.418130    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6567c1f-021d-488b-8e11-8890cd9a8152-registry-tls podName:c6567c1f-021d-488b-8e11-8890cd9a8152 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:02.918109436 +0000 UTC m=+39.769114813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c6567c1f-021d-488b-8e11-8890cd9a8152-registry-tls") pod "image-registry-587f5c755b-xds6m" (UID: "c6567c1f-021d-488b-8e11-8890cd9a8152") : secret "image-registry-tls" not found
Sep 17 18:30:02.418318 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.418194    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6567c1f-021d-488b-8e11-8890cd9a8152-bound-sa-token\") pod \"image-registry-587f5c755b-xds6m\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") " pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:02.418546 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.418366    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c6567c1f-021d-488b-8e11-8890cd9a8152-ca-trust-extracted\") pod \"image-registry-587f5c755b-xds6m\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") " pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:02.418711 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.418666    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c6567c1f-021d-488b-8e11-8890cd9a8152-registry-certificates\") pod \"image-registry-587f5c755b-xds6m\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") " pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:02.419344 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.419321    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6567c1f-021d-488b-8e11-8890cd9a8152-trusted-ca\") pod \"image-registry-587f5c755b-xds6m\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") " pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:02.420925 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.420904    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c6567c1f-021d-488b-8e11-8890cd9a8152-image-registry-private-configuration\") pod \"image-registry-587f5c755b-xds6m\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") " pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:02.421030 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.420909    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c6567c1f-021d-488b-8e11-8890cd9a8152-installation-pull-secrets\") pod \"image-registry-587f5c755b-xds6m\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") " pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:02.429108 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.429086    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6567c1f-021d-488b-8e11-8890cd9a8152-bound-sa-token\") pod \"image-registry-587f5c755b-xds6m\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") " pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:02.429354 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.429334    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlhh7\" (UniqueName: \"kubernetes.io/projected/c6567c1f-021d-488b-8e11-8890cd9a8152-kube-api-access-qlhh7\") pod \"image-registry-587f5c755b-xds6m\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") " pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:02.687428 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.687396    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kpswk" event={"ID":"a7f6c88f-5861-4089-a4a8-75bb85c16c98","Type":"ContainerStarted","Data":"0e23ed7f623ede728086f720d04c35002553c55c862523b56facb6e93e63392f"}
Sep 17 18:30:02.711440 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.711393    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kpswk" podStartSLOduration=9.733173588 podStartE2EDuration="37.711375064s" totalImagesPullingTime="27.978201476s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:02.710543668 +0000 UTC m=+39.561549061" watchObservedRunningTime="2026-09-17 18:30:02.711375064 +0000 UTC m=+39.562380464"
Sep 17 18:30:02.720471 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.720445    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/49a517a5-36f1-4482-b8fc-fa78fb902040-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-j5s56\" (UID: \"49a517a5-36f1-4482-b8fc-fa78fb902040\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-j5s56"
Sep 17 18:30:02.720630 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:02.720612    2588 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 18:30:02.720685 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:02.720675    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49a517a5-36f1-4482-b8fc-fa78fb902040-cluster-monitoring-operator-tls podName:49a517a5-36f1-4482-b8fc-fa78fb902040 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:03.720659042 +0000 UTC m=+40.571664426 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/49a517a5-36f1-4482-b8fc-fa78fb902040-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-j5s56" (UID: "49a517a5-36f1-4482-b8fc-fa78fb902040") : secret "cluster-monitoring-operator-tls" not found
Sep 17 18:30:02.821422 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.821385    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d60900b-84cb-40d3-9ece-74f02821725d-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-p8xqn\" (UID: \"9d60900b-84cb-40d3-9ece-74f02821725d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8xqn"
Sep 17 18:30:02.821590 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:02.821554    2588 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 18:30:02.821652 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:02.821630    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d60900b-84cb-40d3-9ece-74f02821725d-samples-operator-tls podName:9d60900b-84cb-40d3-9ece-74f02821725d nodeName:}" failed. No retries permitted until 2026-09-17 18:30:03.82160032 +0000 UTC m=+40.672605699 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9d60900b-84cb-40d3-9ece-74f02821725d-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-p8xqn" (UID: "9d60900b-84cb-40d3-9ece-74f02821725d") : secret "samples-operator-tls" not found
Sep 17 18:30:02.844742 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.844712    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:02.861545 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.861503    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-x69bl"]
Sep 17 18:30:02.868622 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.868598    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-cf8bw"]
Sep 17 18:30:02.869769 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:30:02.869738    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96541b99_da29_416c_916f_91fbb7207ef1.slice/crio-c1d168bc9cae27e3bbac14b87ce363463e2d763546c0d5c790723ccede0eca7a WatchSource:0}: Error finding container c1d168bc9cae27e3bbac14b87ce363463e2d763546c0d5c790723ccede0eca7a: Status 404 returned error can't find the container with id c1d168bc9cae27e3bbac14b87ce363463e2d763546c0d5c790723ccede0eca7a
Sep 17 18:30:02.873585 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:30:02.873560    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63716ecc_bcb1_44d0_96c4_40ac49599a79.slice/crio-77f77d6997a6935d165dad79142b412f1673f9fc109b02c852fbbb0cf01b24c0 WatchSource:0}: Error finding container 77f77d6997a6935d165dad79142b412f1673f9fc109b02c852fbbb0cf01b24c0: Status 404 returned error can't find the container with id 77f77d6997a6935d165dad79142b412f1673f9fc109b02c852fbbb0cf01b24c0
Sep 17 18:30:02.876626 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.876607    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-xlklb"]
Sep 17 18:30:02.879887 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:30:02.879862    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07ee31e0_43bb_4940_837d_969bd840492b.slice/crio-bea95d3f5aa8a10e13c91a609fefa6f690b073f24093ef85ee94a3be78ba0bc4 WatchSource:0}: Error finding container bea95d3f5aa8a10e13c91a609fefa6f690b073f24093ef85ee94a3be78ba0bc4: Status 404 returned error can't find the container with id bea95d3f5aa8a10e13c91a609fefa6f690b073f24093ef85ee94a3be78ba0bc4
Sep 17 18:30:02.887349 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.887327    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-vsp4l"]
Sep 17 18:30:02.889991 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:30:02.889969    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd80d0a0_6acf_4632_8484_07395506d4c0.slice/crio-800db6ee919a30df62648f5aa5721d30bf2d755223d3b829ab473e91aa406247 WatchSource:0}: Error finding container 800db6ee919a30df62648f5aa5721d30bf2d755223d3b829ab473e91aa406247: Status 404 returned error can't find the container with id 800db6ee919a30df62648f5aa5721d30bf2d755223d3b829ab473e91aa406247
Sep 17 18:30:02.922840 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.922817    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6567c1f-021d-488b-8e11-8890cd9a8152-registry-tls\") pod \"image-registry-587f5c755b-xds6m\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") " pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:02.922936 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:02.922917    2588 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:30:02.922936 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:02.922932    2588 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-587f5c755b-xds6m: secret "image-registry-tls" not found
Sep 17 18:30:02.923002 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:02.922982    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6567c1f-021d-488b-8e11-8890cd9a8152-registry-tls podName:c6567c1f-021d-488b-8e11-8890cd9a8152 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:03.922966955 +0000 UTC m=+40.773972332 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c6567c1f-021d-488b-8e11-8890cd9a8152-registry-tls") pod "image-registry-587f5c755b-xds6m" (UID: "c6567c1f-021d-488b-8e11-8890cd9a8152") : secret "image-registry-tls" not found
Sep 17 18:30:02.935630 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:02.935601    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:03.059660 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:03.059460    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:03.180467 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:03.180391    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:03.326293 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:03.326258    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a7075b4-9ee1-4495-86e5-b2377d9b1b06-metrics-tls\") pod \"dns-default-jq7kv\" (UID: \"6a7075b4-9ee1-4495-86e5-b2377d9b1b06\") " pod="openshift-dns/dns-default-jq7kv"
Sep 17 18:30:03.326439 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:03.326412    2588 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:30:03.326486 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:03.326476    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a7075b4-9ee1-4495-86e5-b2377d9b1b06-metrics-tls podName:6a7075b4-9ee1-4495-86e5-b2377d9b1b06 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:07.326461161 +0000 UTC m=+44.177466538 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a7075b4-9ee1-4495-86e5-b2377d9b1b06-metrics-tls") pod "dns-default-jq7kv" (UID: "6a7075b4-9ee1-4495-86e5-b2377d9b1b06") : secret "dns-default-metrics-tls" not found
Sep 17 18:30:03.691054 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:03.691011    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-x69bl" event={"ID":"96541b99-da29-416c-916f-91fbb7207ef1","Type":"ContainerStarted","Data":"c1d168bc9cae27e3bbac14b87ce363463e2d763546c0d5c790723ccede0eca7a"}
Sep 17 18:30:03.692798 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:03.692767    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6llb8" event={"ID":"676d0454-70c3-4491-b9bc-7d934002b6be","Type":"ContainerStarted","Data":"f4f5d3d9a585ac64433259c94c094daf83a67b8ed40c8028ce2b3db89765370d"}
Sep 17 18:30:03.694107 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:03.694078    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-vsp4l" event={"ID":"fd80d0a0-6acf-4632-8484-07395506d4c0","Type":"ContainerStarted","Data":"800db6ee919a30df62648f5aa5721d30bf2d755223d3b829ab473e91aa406247"}
Sep 17 18:30:03.695230 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:03.695206    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-xlklb" event={"ID":"07ee31e0-43bb-4940-837d-969bd840492b","Type":"ContainerStarted","Data":"bea95d3f5aa8a10e13c91a609fefa6f690b073f24093ef85ee94a3be78ba0bc4"}
Sep 17 18:30:03.696522 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:03.696492    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-cf8bw" event={"ID":"63716ecc-bcb1-44d0-96c4-40ac49599a79","Type":"ContainerStarted","Data":"77f77d6997a6935d165dad79142b412f1673f9fc109b02c852fbbb0cf01b24c0"}
Sep 17 18:30:03.709575 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:03.709515    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6llb8" podStartSLOduration=2.070930184 podStartE2EDuration="4.709499048s" totalImagesPullingTime="2.638568864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:03.708253622 +0000 UTC m=+40.559259021" watchObservedRunningTime="2026-09-17 18:30:03.709499048 +0000 UTC m=+40.560504447"
Sep 17 18:30:03.730841 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:03.730809    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/49a517a5-36f1-4482-b8fc-fa78fb902040-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-j5s56\" (UID: \"49a517a5-36f1-4482-b8fc-fa78fb902040\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-j5s56"
Sep 17 18:30:03.730988 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:03.730954    2588 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 18:30:03.731047 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:03.731034    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49a517a5-36f1-4482-b8fc-fa78fb902040-cluster-monitoring-operator-tls podName:49a517a5-36f1-4482-b8fc-fa78fb902040 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:05.731011288 +0000 UTC m=+42.582016682 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/49a517a5-36f1-4482-b8fc-fa78fb902040-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-j5s56" (UID: "49a517a5-36f1-4482-b8fc-fa78fb902040") : secret "cluster-monitoring-operator-tls" not found
Sep 17 18:30:03.832346 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:03.832308    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d60900b-84cb-40d3-9ece-74f02821725d-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-p8xqn\" (UID: \"9d60900b-84cb-40d3-9ece-74f02821725d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8xqn"
Sep 17 18:30:03.832557 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:03.832511    2588 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 18:30:03.832632 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:03.832609    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d60900b-84cb-40d3-9ece-74f02821725d-samples-operator-tls podName:9d60900b-84cb-40d3-9ece-74f02821725d nodeName:}" failed. No retries permitted until 2026-09-17 18:30:05.832585465 +0000 UTC m=+42.683590859 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9d60900b-84cb-40d3-9ece-74f02821725d-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-p8xqn" (UID: "9d60900b-84cb-40d3-9ece-74f02821725d") : secret "samples-operator-tls" not found
Sep 17 18:30:03.933679 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:03.933644    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6567c1f-021d-488b-8e11-8890cd9a8152-registry-tls\") pod \"image-registry-587f5c755b-xds6m\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") " pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:03.933856 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:03.933826    2588 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:30:03.933856 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:03.933848    2588 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-587f5c755b-xds6m: secret "image-registry-tls" not found
Sep 17 18:30:03.933972 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:03.933922    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6567c1f-021d-488b-8e11-8890cd9a8152-registry-tls podName:c6567c1f-021d-488b-8e11-8890cd9a8152 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:05.933897364 +0000 UTC m=+42.784902761 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c6567c1f-021d-488b-8e11-8890cd9a8152-registry-tls") pod "image-registry-587f5c755b-xds6m" (UID: "c6567c1f-021d-488b-8e11-8890cd9a8152") : secret "image-registry-tls" not found
Sep 17 18:30:03.981637 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:03.981452    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:04.079340 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:04.079303    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:04.702429 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:04.702393    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-x69bl" event={"ID":"96541b99-da29-416c-916f-91fbb7207ef1","Type":"ContainerStarted","Data":"d4b0900e89efa30010e4f91b6d658628f09b1872b9e866e0e841551afebe21be"}
Sep 17 18:30:04.718688 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:04.718307    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-x69bl" podStartSLOduration=2.7497117920000003 podStartE2EDuration="3.718283563s" totalImagesPullingTime="968.571771ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:04.716910547 +0000 UTC m=+41.567915958" watchObservedRunningTime="2026-09-17 18:30:04.718283563 +0000 UTC m=+41.569288963"
Sep 17 18:30:04.821700 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:04.821666    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:04.922503 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:04.922461    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:05.750495 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:05.750454    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/49a517a5-36f1-4482-b8fc-fa78fb902040-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-j5s56\" (UID: \"49a517a5-36f1-4482-b8fc-fa78fb902040\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-j5s56"
Sep 17 18:30:05.750944 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:05.750631    2588 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 18:30:05.750944 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:05.750708    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49a517a5-36f1-4482-b8fc-fa78fb902040-cluster-monitoring-operator-tls podName:49a517a5-36f1-4482-b8fc-fa78fb902040 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:09.750690822 +0000 UTC m=+46.601696219 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/49a517a5-36f1-4482-b8fc-fa78fb902040-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-j5s56" (UID: "49a517a5-36f1-4482-b8fc-fa78fb902040") : secret "cluster-monitoring-operator-tls" not found
Sep 17 18:30:05.851317 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:05.851280    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d60900b-84cb-40d3-9ece-74f02821725d-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-p8xqn\" (UID: \"9d60900b-84cb-40d3-9ece-74f02821725d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8xqn"
Sep 17 18:30:05.851498 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:05.851471    2588 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 18:30:05.851605 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:05.851568    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d60900b-84cb-40d3-9ece-74f02821725d-samples-operator-tls podName:9d60900b-84cb-40d3-9ece-74f02821725d nodeName:}" failed. No retries permitted until 2026-09-17 18:30:09.851545366 +0000 UTC m=+46.702550757 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9d60900b-84cb-40d3-9ece-74f02821725d-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-p8xqn" (UID: "9d60900b-84cb-40d3-9ece-74f02821725d") : secret "samples-operator-tls" not found
Sep 17 18:30:05.952022 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:05.951973    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6567c1f-021d-488b-8e11-8890cd9a8152-registry-tls\") pod \"image-registry-587f5c755b-xds6m\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") " pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:05.952197 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:05.952098    2588 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:30:05.952197 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:05.952116    2588 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-587f5c755b-xds6m: secret "image-registry-tls" not found
Sep 17 18:30:05.952197 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:05.952183    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6567c1f-021d-488b-8e11-8890cd9a8152-registry-tls podName:c6567c1f-021d-488b-8e11-8890cd9a8152 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:09.952164019 +0000 UTC m=+46.803169398 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c6567c1f-021d-488b-8e11-8890cd9a8152-registry-tls") pod "image-registry-587f5c755b-xds6m" (UID: "c6567c1f-021d-488b-8e11-8890cd9a8152") : secret "image-registry-tls" not found
Sep 17 18:30:05.952700 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:05.952678    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:06.052442 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:06.052410    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:06.264503 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:06.264475    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:06.356403 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:06.356330    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:06.456775 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:06.456742    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:06.557397 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:06.557366    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:06.641644 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:06.641565    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:06.980954 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:06.980922    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:07.090242 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:07.090205    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:07.241286 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:07.241201    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:07.342351 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:07.342313    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:07.364254 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:07.364224    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a7075b4-9ee1-4495-86e5-b2377d9b1b06-metrics-tls\") pod \"dns-default-jq7kv\" (UID: \"6a7075b4-9ee1-4495-86e5-b2377d9b1b06\") " pod="openshift-dns/dns-default-jq7kv"
Sep 17 18:30:07.364393 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:07.364376    2588 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:30:07.364450 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:07.364437    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a7075b4-9ee1-4495-86e5-b2377d9b1b06-metrics-tls podName:6a7075b4-9ee1-4495-86e5-b2377d9b1b06 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:15.364419343 +0000 UTC m=+52.215424731 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a7075b4-9ee1-4495-86e5-b2377d9b1b06-metrics-tls") pod "dns-default-jq7kv" (UID: "6a7075b4-9ee1-4495-86e5-b2377d9b1b06") : secret "dns-default-metrics-tls" not found
Sep 17 18:30:07.718581 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:07.718538    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-vsp4l" event={"ID":"fd80d0a0-6acf-4632-8484-07395506d4c0","Type":"ContainerStarted","Data":"b97f17a5850611326ea55c73d343721e30295e9e60fe3273803ddb08d254408c"}
Sep 17 18:30:07.720307 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:07.720283    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xlklb_07ee31e0-43bb-4940-837d-969bd840492b/console-operator/0.log"
Sep 17 18:30:07.720418 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:07.720325    2588 generic.go:356] "Generic (PLEG): container finished" podID="07ee31e0-43bb-4940-837d-969bd840492b" containerID="2594ec38bb1e9a5cb899c0a82abe982c93ca212167d4ca6f1f9806c93f8d2a07" exitCode=255
Sep 17 18:30:07.720418 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:07.720403    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-xlklb" event={"ID":"07ee31e0-43bb-4940-837d-969bd840492b","Type":"ContainerDied","Data":"2594ec38bb1e9a5cb899c0a82abe982c93ca212167d4ca6f1f9806c93f8d2a07"}
Sep 17 18:30:07.720602 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:07.720584    2588 scope.go:117] "RemoveContainer" containerID="2594ec38bb1e9a5cb899c0a82abe982c93ca212167d4ca6f1f9806c93f8d2a07"
Sep 17 18:30:07.722257 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:07.722229    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-cf8bw" event={"ID":"63716ecc-bcb1-44d0-96c4-40ac49599a79","Type":"ContainerStarted","Data":"2f62a541a0f59ce58ab499cce5014c9ca5ef0078f531412a87ed78059e60a812"}
Sep 17 18:30:07.734153 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:07.734108    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-vsp4l" podStartSLOduration=2.715101559 podStartE2EDuration="6.734092835s" totalImagesPullingTime="4.018991276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:07.733507189 +0000 UTC m=+44.584512587" watchObservedRunningTime="2026-09-17 18:30:07.734092835 +0000 UTC m=+44.585098231"
Sep 17 18:30:07.771580 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:07.771516    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-cf8bw" podStartSLOduration=1.7515770609999999 podStartE2EDuration="5.771493037s" totalImagesPullingTime="4.019915976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:07.769461127 +0000 UTC m=+44.620466525" watchObservedRunningTime="2026-09-17 18:30:07.771493037 +0000 UTC m=+44.622498437"
Sep 17 18:30:08.438209 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.438180    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-w9vq7"]
Sep 17 18:30:08.460619 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.457984    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:08.461187 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.461164    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w9vq7"
Sep 17 18:30:08.462823 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.462801    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w9vq7"]
Sep 17 18:30:08.465799 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.465773    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 18:30:08.465922 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.465822    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 18:30:08.465922 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.465850    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-h9v8g\""
Sep 17 18:30:08.574889 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.574850    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/904b068f-aa14-406f-835c-bf966bb4c9ff-crio-socket\") pod \"insights-runtime-extractor-w9vq7\" (UID: \"904b068f-aa14-406f-835c-bf966bb4c9ff\") " pod="openshift-insights/insights-runtime-extractor-w9vq7"
Sep 17 18:30:08.574889 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.574889    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/904b068f-aa14-406f-835c-bf966bb4c9ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9vq7\" (UID: \"904b068f-aa14-406f-835c-bf966bb4c9ff\") " pod="openshift-insights/insights-runtime-extractor-w9vq7"
Sep 17 18:30:08.575088 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.575029    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/904b068f-aa14-406f-835c-bf966bb4c9ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9vq7\" (UID: \"904b068f-aa14-406f-835c-bf966bb4c9ff\") " pod="openshift-insights/insights-runtime-extractor-w9vq7"
Sep 17 18:30:08.575088 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.575073    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9nfk\" (UniqueName: \"kubernetes.io/projected/904b068f-aa14-406f-835c-bf966bb4c9ff-kube-api-access-n9nfk\") pod \"insights-runtime-extractor-w9vq7\" (UID: \"904b068f-aa14-406f-835c-bf966bb4c9ff\") " pod="openshift-insights/insights-runtime-extractor-w9vq7"
Sep 17 18:30:08.575161 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.575093    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/904b068f-aa14-406f-835c-bf966bb4c9ff-data-volume\") pod \"insights-runtime-extractor-w9vq7\" (UID: \"904b068f-aa14-406f-835c-bf966bb4c9ff\") " pod="openshift-insights/insights-runtime-extractor-w9vq7"
Sep 17 18:30:08.581626 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.581597    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:08.675654 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.675616    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/904b068f-aa14-406f-835c-bf966bb4c9ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9vq7\" (UID: \"904b068f-aa14-406f-835c-bf966bb4c9ff\") " pod="openshift-insights/insights-runtime-extractor-w9vq7"
Sep 17 18:30:08.675795 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.675722    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/904b068f-aa14-406f-835c-bf966bb4c9ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9vq7\" (UID: \"904b068f-aa14-406f-835c-bf966bb4c9ff\") " pod="openshift-insights/insights-runtime-extractor-w9vq7"
Sep 17 18:30:08.675795 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:08.675738    2588 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:30:08.675795 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.675757    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9nfk\" (UniqueName: \"kubernetes.io/projected/904b068f-aa14-406f-835c-bf966bb4c9ff-kube-api-access-n9nfk\") pod \"insights-runtime-extractor-w9vq7\" (UID: \"904b068f-aa14-406f-835c-bf966bb4c9ff\") " pod="openshift-insights/insights-runtime-extractor-w9vq7"
Sep 17 18:30:08.675795 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.675774    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/904b068f-aa14-406f-835c-bf966bb4c9ff-data-volume\") pod \"insights-runtime-extractor-w9vq7\" (UID: \"904b068f-aa14-406f-835c-bf966bb4c9ff\") " pod="openshift-insights/insights-runtime-extractor-w9vq7"
Sep 17 18:30:08.676020 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:08.675809    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/904b068f-aa14-406f-835c-bf966bb4c9ff-kube-rbac-proxy-cm podName:904b068f-aa14-406f-835c-bf966bb4c9ff nodeName:}" failed. No retries permitted until 2026-09-17 18:30:09.175787126 +0000 UTC m=+46.026792502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/904b068f-aa14-406f-835c-bf966bb4c9ff-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w9vq7" (UID: "904b068f-aa14-406f-835c-bf966bb4c9ff") : configmap "kube-rbac-proxy" not found
Sep 17 18:30:08.676020 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:08.675878    2588 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:08.676020 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.675886    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/904b068f-aa14-406f-835c-bf966bb4c9ff-crio-socket\") pod \"insights-runtime-extractor-w9vq7\" (UID: \"904b068f-aa14-406f-835c-bf966bb4c9ff\") " pod="openshift-insights/insights-runtime-extractor-w9vq7"
Sep 17 18:30:08.676020 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:08.675950    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/904b068f-aa14-406f-835c-bf966bb4c9ff-insights-runtime-extractor-tls podName:904b068f-aa14-406f-835c-bf966bb4c9ff nodeName:}" failed. No retries permitted until 2026-09-17 18:30:09.175933245 +0000 UTC m=+46.026938628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/904b068f-aa14-406f-835c-bf966bb4c9ff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w9vq7" (UID: "904b068f-aa14-406f-835c-bf966bb4c9ff") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:08.676187 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.676068    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/904b068f-aa14-406f-835c-bf966bb4c9ff-crio-socket\") pod \"insights-runtime-extractor-w9vq7\" (UID: \"904b068f-aa14-406f-835c-bf966bb4c9ff\") " pod="openshift-insights/insights-runtime-extractor-w9vq7"
Sep 17 18:30:08.676187 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.676077    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/904b068f-aa14-406f-835c-bf966bb4c9ff-data-volume\") pod \"insights-runtime-extractor-w9vq7\" (UID: \"904b068f-aa14-406f-835c-bf966bb4c9ff\") " pod="openshift-insights/insights-runtime-extractor-w9vq7"
Sep 17 18:30:08.712318 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.712249    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9nfk\" (UniqueName: \"kubernetes.io/projected/904b068f-aa14-406f-835c-bf966bb4c9ff-kube-api-access-n9nfk\") pod \"insights-runtime-extractor-w9vq7\" (UID: \"904b068f-aa14-406f-835c-bf966bb4c9ff\") " pod="openshift-insights/insights-runtime-extractor-w9vq7"
Sep 17 18:30:08.727432 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.727405    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xlklb_07ee31e0-43bb-4940-837d-969bd840492b/console-operator/1.log"
Sep 17 18:30:08.727833 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.727818    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xlklb_07ee31e0-43bb-4940-837d-969bd840492b/console-operator/0.log"
Sep 17 18:30:08.727878 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.727856    2588 generic.go:356] "Generic (PLEG): container finished" podID="07ee31e0-43bb-4940-837d-969bd840492b" containerID="667b501b2c9d8f53553b0d0ce3d47f0733c5d312685ba8010b619fc4e2fdd7b7" exitCode=255
Sep 17 18:30:08.727964 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.727941    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-xlklb" event={"ID":"07ee31e0-43bb-4940-837d-969bd840492b","Type":"ContainerDied","Data":"667b501b2c9d8f53553b0d0ce3d47f0733c5d312685ba8010b619fc4e2fdd7b7"}
Sep 17 18:30:08.728013 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.727999    2588 scope.go:117] "RemoveContainer" containerID="2594ec38bb1e9a5cb899c0a82abe982c93ca212167d4ca6f1f9806c93f8d2a07"
Sep 17 18:30:08.728313 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.728296    2588 scope.go:117] "RemoveContainer" containerID="667b501b2c9d8f53553b0d0ce3d47f0733c5d312685ba8010b619fc4e2fdd7b7"
Sep 17 18:30:08.728494 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:08.728476    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-xlklb_openshift-console-operator(07ee31e0-43bb-4940-837d-969bd840492b)\"" pod="openshift-console-operator/console-operator-fd4bb46c-xlklb" podUID="07ee31e0-43bb-4940-837d-969bd840492b"
Sep 17 18:30:08.842788 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.842753    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-qh5gq"]
Sep 17 18:30:08.845575 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.845557    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-qh5gq"
Sep 17 18:30:08.848157 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.848134    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 17 18:30:08.848157 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.848158    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 17 18:30:08.848336 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.848159    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 17 18:30:08.848336 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.848181    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-kwr4f\""
Sep 17 18:30:08.848336 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.848240    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 17 18:30:08.855350 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.855328    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-qh5gq"]
Sep 17 18:30:08.909390 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.909361    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:08.971002 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.970923    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:08.979102 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.979082    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ea3202b3-4572-413c-be84-aacc00d35f75-signing-key\") pod \"service-ca-55f6fc7b86-qh5gq\" (UID: \"ea3202b3-4572-413c-be84-aacc00d35f75\") " pod="openshift-service-ca/service-ca-55f6fc7b86-qh5gq"
Sep 17 18:30:08.979166 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.979145    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cl9s\" (UniqueName: \"kubernetes.io/projected/ea3202b3-4572-413c-be84-aacc00d35f75-kube-api-access-8cl9s\") pod \"service-ca-55f6fc7b86-qh5gq\" (UID: \"ea3202b3-4572-413c-be84-aacc00d35f75\") " pod="openshift-service-ca/service-ca-55f6fc7b86-qh5gq"
Sep 17 18:30:08.979204 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:08.979190    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ea3202b3-4572-413c-be84-aacc00d35f75-signing-cabundle\") pod \"service-ca-55f6fc7b86-qh5gq\" (UID: \"ea3202b3-4572-413c-be84-aacc00d35f75\") " pod="openshift-service-ca/service-ca-55f6fc7b86-qh5gq"
Sep 17 18:30:09.082203 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:09.082165    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ea3202b3-4572-413c-be84-aacc00d35f75-signing-key\") pod \"service-ca-55f6fc7b86-qh5gq\" (UID: \"ea3202b3-4572-413c-be84-aacc00d35f75\") " pod="openshift-service-ca/service-ca-55f6fc7b86-qh5gq"
Sep 17 18:30:09.082394 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:09.082242    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cl9s\" (UniqueName: \"kubernetes.io/projected/ea3202b3-4572-413c-be84-aacc00d35f75-kube-api-access-8cl9s\") pod \"service-ca-55f6fc7b86-qh5gq\" (UID: \"ea3202b3-4572-413c-be84-aacc00d35f75\") " pod="openshift-service-ca/service-ca-55f6fc7b86-qh5gq"
Sep 17 18:30:09.082394 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:09.082283    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ea3202b3-4572-413c-be84-aacc00d35f75-signing-cabundle\") pod \"service-ca-55f6fc7b86-qh5gq\" (UID: \"ea3202b3-4572-413c-be84-aacc00d35f75\") " pod="openshift-service-ca/service-ca-55f6fc7b86-qh5gq"
Sep 17 18:30:09.083033 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:09.083011    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:09.083214 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:09.083193    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ea3202b3-4572-413c-be84-aacc00d35f75-signing-cabundle\") pod \"service-ca-55f6fc7b86-qh5gq\" (UID: \"ea3202b3-4572-413c-be84-aacc00d35f75\") " pod="openshift-service-ca/service-ca-55f6fc7b86-qh5gq"
Sep 17 18:30:09.084765 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:09.084744    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ea3202b3-4572-413c-be84-aacc00d35f75-signing-key\") pod \"service-ca-55f6fc7b86-qh5gq\" (UID: \"ea3202b3-4572-413c-be84-aacc00d35f75\") " pod="openshift-service-ca/service-ca-55f6fc7b86-qh5gq"
Sep 17 18:30:09.101882 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:09.101852    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cl9s\" (UniqueName: \"kubernetes.io/projected/ea3202b3-4572-413c-be84-aacc00d35f75-kube-api-access-8cl9s\") pod \"service-ca-55f6fc7b86-qh5gq\" (UID: \"ea3202b3-4572-413c-be84-aacc00d35f75\") " pod="openshift-service-ca/service-ca-55f6fc7b86-qh5gq"
Sep 17 18:30:09.154580 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:09.154548    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-qh5gq"
Sep 17 18:30:09.182913 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:09.182884    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/904b068f-aa14-406f-835c-bf966bb4c9ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9vq7\" (UID: \"904b068f-aa14-406f-835c-bf966bb4c9ff\") " pod="openshift-insights/insights-runtime-extractor-w9vq7"
Sep 17 18:30:09.183011 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:09.182980    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/904b068f-aa14-406f-835c-bf966bb4c9ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9vq7\" (UID: \"904b068f-aa14-406f-835c-bf966bb4c9ff\") " pod="openshift-insights/insights-runtime-extractor-w9vq7"
Sep 17 18:30:09.183059 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:09.183004    2588 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:30:09.183094 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:09.183080    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/904b068f-aa14-406f-835c-bf966bb4c9ff-kube-rbac-proxy-cm podName:904b068f-aa14-406f-835c-bf966bb4c9ff nodeName:}" failed. No retries permitted until 2026-09-17 18:30:10.183054085 +0000 UTC m=+47.034059466 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/904b068f-aa14-406f-835c-bf966bb4c9ff-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w9vq7" (UID: "904b068f-aa14-406f-835c-bf966bb4c9ff") : configmap "kube-rbac-proxy" not found
Sep 17 18:30:09.183140 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:09.183122    2588 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:09.183193 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:09.183182    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/904b068f-aa14-406f-835c-bf966bb4c9ff-insights-runtime-extractor-tls podName:904b068f-aa14-406f-835c-bf966bb4c9ff nodeName:}" failed. No retries permitted until 2026-09-17 18:30:10.183168784 +0000 UTC m=+47.034174175 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/904b068f-aa14-406f-835c-bf966bb4c9ff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w9vq7" (UID: "904b068f-aa14-406f-835c-bf966bb4c9ff") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:09.187396 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:09.187369    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:09.428644 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:09.428509    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-qh5gq"]
Sep 17 18:30:09.431118 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:30:09.431089    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea3202b3_4572_413c_be84_aacc00d35f75.slice/crio-c0a1feef72775a679c9185bff5b63da6ad581ecfaa2cac7fdcf01332d68dc9be WatchSource:0}: Error finding container c0a1feef72775a679c9185bff5b63da6ad581ecfaa2cac7fdcf01332d68dc9be: Status 404 returned error can't find the container with id c0a1feef72775a679c9185bff5b63da6ad581ecfaa2cac7fdcf01332d68dc9be
Sep 17 18:30:09.734868 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:09.734786    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-qh5gq" event={"ID":"ea3202b3-4572-413c-be84-aacc00d35f75","Type":"ContainerStarted","Data":"c0a1feef72775a679c9185bff5b63da6ad581ecfaa2cac7fdcf01332d68dc9be"}
Sep 17 18:30:09.736341 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:09.736321    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xlklb_07ee31e0-43bb-4940-837d-969bd840492b/console-operator/1.log"
Sep 17 18:30:09.736683 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:09.736670    2588 scope.go:117] "RemoveContainer" containerID="667b501b2c9d8f53553b0d0ce3d47f0733c5d312685ba8010b619fc4e2fdd7b7"
Sep 17 18:30:09.736837 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:09.736820    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-xlklb_openshift-console-operator(07ee31e0-43bb-4940-837d-969bd840492b)\"" pod="openshift-console-operator/console-operator-fd4bb46c-xlklb" podUID="07ee31e0-43bb-4940-837d-969bd840492b"
Sep 17 18:30:09.788134 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:09.788104    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/49a517a5-36f1-4482-b8fc-fa78fb902040-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-j5s56\" (UID: \"49a517a5-36f1-4482-b8fc-fa78fb902040\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-j5s56"
Sep 17 18:30:09.788285 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:09.788267    2588 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 18:30:09.788344 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:09.788332    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49a517a5-36f1-4482-b8fc-fa78fb902040-cluster-monitoring-operator-tls podName:49a517a5-36f1-4482-b8fc-fa78fb902040 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:17.788315851 +0000 UTC m=+54.639321228 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/49a517a5-36f1-4482-b8fc-fa78fb902040-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-j5s56" (UID: "49a517a5-36f1-4482-b8fc-fa78fb902040") : secret "cluster-monitoring-operator-tls" not found
Sep 17 18:30:09.889060 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:09.889020    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d60900b-84cb-40d3-9ece-74f02821725d-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-p8xqn\" (UID: \"9d60900b-84cb-40d3-9ece-74f02821725d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8xqn"
Sep 17 18:30:09.889234 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:09.889193    2588 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 18:30:09.889305 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:09.889291    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d60900b-84cb-40d3-9ece-74f02821725d-samples-operator-tls podName:9d60900b-84cb-40d3-9ece-74f02821725d nodeName:}" failed. No retries permitted until 2026-09-17 18:30:17.889267607 +0000 UTC m=+54.740273001 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9d60900b-84cb-40d3-9ece-74f02821725d-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-p8xqn" (UID: "9d60900b-84cb-40d3-9ece-74f02821725d") : secret "samples-operator-tls" not found
Sep 17 18:30:09.990462 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:09.990379    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6567c1f-021d-488b-8e11-8890cd9a8152-registry-tls\") pod \"image-registry-587f5c755b-xds6m\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") " pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:09.990641 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:09.990570    2588 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:30:09.990641 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:09.990594    2588 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-587f5c755b-xds6m: secret "image-registry-tls" not found
Sep 17 18:30:09.990726 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:09.990679    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6567c1f-021d-488b-8e11-8890cd9a8152-registry-tls podName:c6567c1f-021d-488b-8e11-8890cd9a8152 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:17.990657614 +0000 UTC m=+54.841663003 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c6567c1f-021d-488b-8e11-8890cd9a8152-registry-tls") pod "image-registry-587f5c755b-xds6m" (UID: "c6567c1f-021d-488b-8e11-8890cd9a8152") : secret "image-registry-tls" not found
Sep 17 18:30:10.041450 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:10.041419    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:10.192755 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:10.192721    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/904b068f-aa14-406f-835c-bf966bb4c9ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9vq7\" (UID: \"904b068f-aa14-406f-835c-bf966bb4c9ff\") " pod="openshift-insights/insights-runtime-extractor-w9vq7"
Sep 17 18:30:10.192930 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:10.192822    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/904b068f-aa14-406f-835c-bf966bb4c9ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9vq7\" (UID: \"904b068f-aa14-406f-835c-bf966bb4c9ff\") " pod="openshift-insights/insights-runtime-extractor-w9vq7"
Sep 17 18:30:10.192930 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:10.192901    2588 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:10.192930 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:10.192918    2588 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:30:10.193074 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:10.192970    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/904b068f-aa14-406f-835c-bf966bb4c9ff-kube-rbac-proxy-cm podName:904b068f-aa14-406f-835c-bf966bb4c9ff nodeName:}" failed. No retries permitted until 2026-09-17 18:30:12.192956949 +0000 UTC m=+49.043962326 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/904b068f-aa14-406f-835c-bf966bb4c9ff-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w9vq7" (UID: "904b068f-aa14-406f-835c-bf966bb4c9ff") : configmap "kube-rbac-proxy" not found
Sep 17 18:30:10.193074 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:10.192983    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/904b068f-aa14-406f-835c-bf966bb4c9ff-insights-runtime-extractor-tls podName:904b068f-aa14-406f-835c-bf966bb4c9ff nodeName:}" failed. No retries permitted until 2026-09-17 18:30:12.192976986 +0000 UTC m=+49.043982363 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/904b068f-aa14-406f-835c-bf966bb4c9ff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w9vq7" (UID: "904b068f-aa14-406f-835c-bf966bb4c9ff") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:10.595272 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:10.595229    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-745b6_61513726-d6fb-4e35-a39f-818bcb1a9596/dns-node-resolver/0.log"
Sep 17 18:30:11.593715 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:11.593691    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5cb86_947781e1-4423-435e-b44f-53ca352f8df7/node-ca/0.log"
Sep 17 18:30:11.746499 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:11.746464    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-qh5gq" event={"ID":"ea3202b3-4572-413c-be84-aacc00d35f75","Type":"ContainerStarted","Data":"60ccb5f5f884aff23cb85773569926fb4532494f80247a56225dfc6cb815b8e0"}
Sep 17 18:30:11.764295 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:11.764251    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-qh5gq" podStartSLOduration=2.092406529 podStartE2EDuration="3.764235925s" totalImagesPullingTime="1.671829396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:11.763196047 +0000 UTC m=+48.614201446" watchObservedRunningTime="2026-09-17 18:30:11.764235925 +0000 UTC m=+48.615241350"
Sep 17 18:30:12.211397 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:12.211358    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/904b068f-aa14-406f-835c-bf966bb4c9ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9vq7\" (UID: \"904b068f-aa14-406f-835c-bf966bb4c9ff\") " pod="openshift-insights/insights-runtime-extractor-w9vq7"
Sep 17 18:30:12.211627 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:12.211476    2588 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:30:12.211627 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:12.211521    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/904b068f-aa14-406f-835c-bf966bb4c9ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9vq7\" (UID: \"904b068f-aa14-406f-835c-bf966bb4c9ff\") " pod="openshift-insights/insights-runtime-extractor-w9vq7"
Sep 17 18:30:12.211627 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:12.211551    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/904b068f-aa14-406f-835c-bf966bb4c9ff-kube-rbac-proxy-cm podName:904b068f-aa14-406f-835c-bf966bb4c9ff nodeName:}" failed. No retries permitted until 2026-09-17 18:30:16.211519461 +0000 UTC m=+53.062524838 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/904b068f-aa14-406f-835c-bf966bb4c9ff-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w9vq7" (UID: "904b068f-aa14-406f-835c-bf966bb4c9ff") : configmap "kube-rbac-proxy" not found
Sep 17 18:30:12.211627 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:12.211613    2588 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:12.211848 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:12.211654    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/904b068f-aa14-406f-835c-bf966bb4c9ff-insights-runtime-extractor-tls podName:904b068f-aa14-406f-835c-bf966bb4c9ff nodeName:}" failed. No retries permitted until 2026-09-17 18:30:16.211643288 +0000 UTC m=+53.062648665 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/904b068f-aa14-406f-835c-bf966bb4c9ff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w9vq7" (UID: "904b068f-aa14-406f-835c-bf966bb4c9ff") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:12.282237 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:12.282200    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-xlklb"
Sep 17 18:30:12.282396 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:12.282253    2588 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-xlklb"
Sep 17 18:30:12.282629 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:12.282615    2588 scope.go:117] "RemoveContainer" containerID="667b501b2c9d8f53553b0d0ce3d47f0733c5d312685ba8010b619fc4e2fdd7b7"
Sep 17 18:30:12.282815 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:12.282796    2588 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-xlklb_openshift-console-operator(07ee31e0-43bb-4940-837d-969bd840492b)\"" pod="openshift-console-operator/console-operator-fd4bb46c-xlklb" podUID="07ee31e0-43bb-4940-837d-969bd840492b"
Sep 17 18:30:12.393902 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:12.393873    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6llb8_676d0454-70c3-4491-b9bc-7d934002b6be/serve-healthcheck-canary/0.log"
Sep 17 18:30:12.636798 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:12.636769    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:12.811294 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:12.811214    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:12.923507 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:12.923385    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:13.031179 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:13.031141    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:13.212950 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:13.212868    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:13.462899 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:13.462870    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:13.564669 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:13.564625    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:13.709584 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:13.709546    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:14.189500 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:14.189467    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:14.512110 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:14.512032    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:14.718503 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:14.718475    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:15.122232 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:15.122195    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:15.436228 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:15.436144    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a7075b4-9ee1-4495-86e5-b2377d9b1b06-metrics-tls\") pod \"dns-default-jq7kv\" (UID: \"6a7075b4-9ee1-4495-86e5-b2377d9b1b06\") " pod="openshift-dns/dns-default-jq7kv"
Sep 17 18:30:15.436363 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:15.436303    2588 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:30:15.436409 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:15.436371    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a7075b4-9ee1-4495-86e5-b2377d9b1b06-metrics-tls podName:6a7075b4-9ee1-4495-86e5-b2377d9b1b06 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:31.436352984 +0000 UTC m=+68.287358360 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a7075b4-9ee1-4495-86e5-b2377d9b1b06-metrics-tls") pod "dns-default-jq7kv" (UID: "6a7075b4-9ee1-4495-86e5-b2377d9b1b06") : secret "dns-default-metrics-tls" not found
Sep 17 18:30:15.711700 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:15.711618    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:16.146427 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:16.146392    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:16.243279 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:16.243249    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/904b068f-aa14-406f-835c-bf966bb4c9ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9vq7\" (UID: \"904b068f-aa14-406f-835c-bf966bb4c9ff\") " pod="openshift-insights/insights-runtime-extractor-w9vq7"
Sep 17 18:30:16.243417 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:16.243340    2588 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:30:16.243417 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:16.243350    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/904b068f-aa14-406f-835c-bf966bb4c9ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9vq7\" (UID: \"904b068f-aa14-406f-835c-bf966bb4c9ff\") " pod="openshift-insights/insights-runtime-extractor-w9vq7"
Sep 17 18:30:16.243496 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:16.243423    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/904b068f-aa14-406f-835c-bf966bb4c9ff-kube-rbac-proxy-cm podName:904b068f-aa14-406f-835c-bf966bb4c9ff nodeName:}" failed. No retries permitted until 2026-09-17 18:30:24.243398843 +0000 UTC m=+61.094404243 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/904b068f-aa14-406f-835c-bf966bb4c9ff-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w9vq7" (UID: "904b068f-aa14-406f-835c-bf966bb4c9ff") : configmap "kube-rbac-proxy" not found
Sep 17 18:30:16.243496 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:16.243468    2588 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:16.243611 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:16.243516    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/904b068f-aa14-406f-835c-bf966bb4c9ff-insights-runtime-extractor-tls podName:904b068f-aa14-406f-835c-bf966bb4c9ff nodeName:}" failed. No retries permitted until 2026-09-17 18:30:24.243501365 +0000 UTC m=+61.094506741 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/904b068f-aa14-406f-835c-bf966bb4c9ff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w9vq7" (UID: "904b068f-aa14-406f-835c-bf966bb4c9ff") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:16.300845 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:16.300811    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:16.491721 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:16.491644    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:16.591540 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:16.591502    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:16.912297 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:16.912266    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:17.306247 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:17.306221    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:17.516798 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:17.516760    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:17.857269 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:17.857231    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/49a517a5-36f1-4482-b8fc-fa78fb902040-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-j5s56\" (UID: \"49a517a5-36f1-4482-b8fc-fa78fb902040\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-j5s56"
Sep 17 18:30:17.857479 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:17.857404    2588 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 18:30:17.857571 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:17.857487    2588 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49a517a5-36f1-4482-b8fc-fa78fb902040-cluster-monitoring-operator-tls podName:49a517a5-36f1-4482-b8fc-fa78fb902040 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:33.857465283 +0000 UTC m=+70.708470669 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/49a517a5-36f1-4482-b8fc-fa78fb902040-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-j5s56" (UID: "49a517a5-36f1-4482-b8fc-fa78fb902040") : secret "cluster-monitoring-operator-tls" not found
Sep 17 18:30:17.958484 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:17.958443    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d60900b-84cb-40d3-9ece-74f02821725d-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-p8xqn\" (UID: \"9d60900b-84cb-40d3-9ece-74f02821725d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8xqn"
Sep 17 18:30:17.960877 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:17.960857    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9d60900b-84cb-40d3-9ece-74f02821725d-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-p8xqn\" (UID: \"9d60900b-84cb-40d3-9ece-74f02821725d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8xqn"
Sep 17 18:30:17.985754 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:17.985723    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8xqn"
Sep 17 18:30:18.059169 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:18.059131    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6567c1f-021d-488b-8e11-8890cd9a8152-registry-tls\") pod \"image-registry-587f5c755b-xds6m\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") " pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:18.061967 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:18.061946    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6567c1f-021d-488b-8e11-8890cd9a8152-registry-tls\") pod \"image-registry-587f5c755b-xds6m\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") " pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:18.101138 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:18.101110    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:18.109369 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:18.109317    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:18.271254 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:18.271224    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8xqn"]
Sep 17 18:30:18.390097 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:18.390040    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-587f5c755b-xds6m"]
Sep 17 18:30:18.393075 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:30:18.393052    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6567c1f_021d_488b_8e11_8890cd9a8152.slice/crio-ae17008b3c49e1460f52025fcc8116694b41338644df7cca4f69c2710b3d15da WatchSource:0}: Error finding container ae17008b3c49e1460f52025fcc8116694b41338644df7cca4f69c2710b3d15da: Status 404 returned error can't find the container with id ae17008b3c49e1460f52025fcc8116694b41338644df7cca4f69c2710b3d15da
Sep 17 18:30:18.707353 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:18.707278    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:18.768954 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:18.768917    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8xqn" event={"ID":"9d60900b-84cb-40d3-9ece-74f02821725d","Type":"ContainerStarted","Data":"77f882ae617b2b6536014cb11b55a6ab9e360ccb1bdbd606697d85c2abb6848e"}
Sep 17 18:30:18.770444 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:18.770422    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-587f5c755b-xds6m" event={"ID":"c6567c1f-021d-488b-8e11-8890cd9a8152","Type":"ContainerStarted","Data":"61a422e2655612a306f5c793298e921237f506596b9ef0eb6fa5510c416ff19b"}
Sep 17 18:30:18.770569 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:18.770463    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-587f5c755b-xds6m" event={"ID":"c6567c1f-021d-488b-8e11-8890cd9a8152","Type":"ContainerStarted","Data":"ae17008b3c49e1460f52025fcc8116694b41338644df7cca4f69c2710b3d15da"}
Sep 17 18:30:18.770569 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:18.770559    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:18.789694 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:18.789634    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-587f5c755b-xds6m" podStartSLOduration=16.789607203 podStartE2EDuration="16.789607203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:18.788736326 +0000 UTC m=+55.639741726" watchObservedRunningTime="2026-09-17 18:30:18.789607203 +0000 UTC m=+55.640612602"
Sep 17 18:30:18.893081 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:18.893041    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:18.947426 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:18.947391    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:19.304687 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:19.304659    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:19.488669 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:19.488634    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:19.549722 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:19.549692    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:19.908430 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:19.908400    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:20.708850 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:20.708773    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:20.780569 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:20.780516    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8xqn" event={"ID":"9d60900b-84cb-40d3-9ece-74f02821725d","Type":"ContainerStarted","Data":"f9f98c01ba5b657433a26481fa120edc1d8eb389742d8569ae36bf1076bb317b"}
Sep 17 18:30:20.780740 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:20.780587    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8xqn" event={"ID":"9d60900b-84cb-40d3-9ece-74f02821725d","Type":"ContainerStarted","Data":"75ffab85b4fa4e139b254ee086506589001a81c95e96bd08997f8ca5c0dc92ea"}
Sep 17 18:30:20.799829 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:20.799774    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-p8xqn" podStartSLOduration=16.825389072 podStartE2EDuration="18.799759336s" totalImagesPullingTime="1.974370264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:20.799344372 +0000 UTC m=+57.650349772" watchObservedRunningTime="2026-09-17 18:30:20.799759336 +0000 UTC m=+57.650764734"
Sep 17 18:30:20.914500 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:20.914463    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:21.316103 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:21.316054    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:21.518234 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:21.518187    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:21.585404 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:21.585325    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:22.113644 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:22.113604    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:22.526000 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:22.525924    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:22.732791 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:22.732762    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:22.915844 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:22.915814    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:23.504973 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:23.504949    2588 scope.go:117] "RemoveContainer" containerID="667b501b2c9d8f53553b0d0ce3d47f0733c5d312685ba8010b619fc4e2fdd7b7"
Sep 17 18:30:23.645274 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:23.645240    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:23.791878 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:23.791809    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xlklb_07ee31e0-43bb-4940-837d-969bd840492b/console-operator/1.log"
Sep 17 18:30:23.792013 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:23.791880    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-xlklb" event={"ID":"07ee31e0-43bb-4940-837d-969bd840492b","Type":"ContainerStarted","Data":"6f56664fba6faae80082607980555aa370c15ea954d2275b9295a1148ad501b4"}
Sep 17 18:30:23.792175 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:23.792152    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-xlklb"
Sep 17 18:30:23.808032 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:23.807945    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-xlklb" podStartSLOduration=18.790173579 podStartE2EDuration="22.807927777s" totalImagesPullingTime="4.017754198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:23.807767702 +0000 UTC m=+60.658773101" watchObservedRunningTime="2026-09-17 18:30:23.807927777 +0000 UTC m=+60.658933177"
Sep 17 18:30:23.998027 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:23.997999    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-xlklb"
Sep 17 18:30:24.180877 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:24.180848    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:24.284363 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:24.284334    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:24.310564 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:24.310539    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/904b068f-aa14-406f-835c-bf966bb4c9ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9vq7\" (UID: \"904b068f-aa14-406f-835c-bf966bb4c9ff\") " pod="openshift-insights/insights-runtime-extractor-w9vq7"
Sep 17 18:30:24.310697 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:24.310682    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/904b068f-aa14-406f-835c-bf966bb4c9ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9vq7\" (UID: \"904b068f-aa14-406f-835c-bf966bb4c9ff\") " pod="openshift-insights/insights-runtime-extractor-w9vq7"
Sep 17 18:30:24.311047 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:24.311031    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/904b068f-aa14-406f-835c-bf966bb4c9ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w9vq7\" (UID: \"904b068f-aa14-406f-835c-bf966bb4c9ff\") " pod="openshift-insights/insights-runtime-extractor-w9vq7"
Sep 17 18:30:24.312968 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:24.312948    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/904b068f-aa14-406f-835c-bf966bb4c9ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w9vq7\" (UID: \"904b068f-aa14-406f-835c-bf966bb4c9ff\") " pod="openshift-insights/insights-runtime-extractor-w9vq7"
Sep 17 18:30:24.371120 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:24.371091    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w9vq7"
Sep 17 18:30:24.384117 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:24.384091    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:24.482470 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:24.482439    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:24.667053 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:24.667023    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w9vq7"]
Sep 17 18:30:24.668714 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:24.668693    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2rqhq"
Sep 17 18:30:24.670024 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:30:24.669996    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod904b068f_aa14_406f_835c_bf966bb4c9ff.slice/crio-e033fb8135d26aafdefdf0254a28119cae2380584057e9ec2f63807eab2917df WatchSource:0}: Error finding container e033fb8135d26aafdefdf0254a28119cae2380584057e9ec2f63807eab2917df: Status 404 returned error can't find the container with id e033fb8135d26aafdefdf0254a28119cae2380584057e9ec2f63807eab2917df
Sep 17 18:30:24.796641 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:24.796617    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w9vq7" event={"ID":"904b068f-aa14-406f-835c-bf966bb4c9ff","Type":"ContainerStarted","Data":"e033fb8135d26aafdefdf0254a28119cae2380584057e9ec2f63807eab2917df"}
Sep 17 18:30:24.876274 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:24.876239    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:24.950862 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:24.950828    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:25.078265 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:25.078239    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:25.147301 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:25.147264    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:25.270885 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:25.270856    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:25.371377 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:25.371308    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:25.801094 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:25.801055    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w9vq7" event={"ID":"904b068f-aa14-406f-835c-bf966bb4c9ff","Type":"ContainerStarted","Data":"c246b9ed62556c525e1df99c530f96ef0b78863e5dbe7a28042e3648a5591bf0"}
Sep 17 18:30:25.870903 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:25.870876    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:26.067628 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:26.067521    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:26.336113 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:26.336032    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:26.806833 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:26.806793    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w9vq7" event={"ID":"904b068f-aa14-406f-835c-bf966bb4c9ff","Type":"ContainerStarted","Data":"a7c10580ba45306c9dad5fd065e836466b5ed8440a68482498460a25f3e81a8c"}
Sep 17 18:30:26.887211 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:26.887177    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:26.988055 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:26.987997    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:27.632784 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:27.632755    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:27.727610 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:27.727580    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:28.819983 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:28.819945    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w9vq7" event={"ID":"904b068f-aa14-406f-835c-bf966bb4c9ff","Type":"ContainerStarted","Data":"a6be711852bbc95f1b5a10ac79da168ad34f0da3d10bf2bb37ca17fe2641f55b"}
Sep 17 18:30:28.838383 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:28.838338    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-w9vq7" podStartSLOduration=17.951064903 podStartE2EDuration="20.838321988s" totalImagesPullingTime="2.887257085s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:30:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:28.83667982 +0000 UTC m=+65.687685219" watchObservedRunningTime="2026-09-17 18:30:28.838321988 +0000 UTC m=+65.689327388"
Sep 17 18:30:29.834335 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:29.834303    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b89b77694-4hs2h"]
Sep 17 18:30:29.836607 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:29.836588    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b89b77694-4hs2h"
Sep 17 18:30:29.840916 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:29.840892    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 18:30:29.841596 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:29.841128    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 18:30:29.841596 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:29.841348    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 18:30:29.841596 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:29.841577    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-w2sp5\""
Sep 17 18:30:29.841967 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:29.841702    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 18:30:29.842975 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:29.842951    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775568978-sxkzc"]
Sep 17 18:30:29.845593 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:29.845571    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775568978-sxkzc"
Sep 17 18:30:29.848042 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:29.848021    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 18:30:29.849170 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:29.849129    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 18:30:29.849170 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:29.849163    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 18:30:29.852178 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:29.852154    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 18:30:29.858308 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:29.858283    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b89b77694-4hs2h"]
Sep 17 18:30:29.884958 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:29.884933    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775568978-sxkzc"]
Sep 17 18:30:29.900485 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:29.900454    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-587f5c755b-xds6m"]
Sep 17 18:30:29.960576 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:29.960518    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/14850890-1e36-4ab7-b480-516b4d81655e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5b89b77694-4hs2h\" (UID: \"14850890-1e36-4ab7-b480-516b4d81655e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b89b77694-4hs2h"
Sep 17 18:30:29.960762 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:29.960620    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d77fa865-aa05-4f6b-bf94-fceea608785a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-775568978-sxkzc\" (UID: \"d77fa865-aa05-4f6b-bf94-fceea608785a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775568978-sxkzc"
Sep 17 18:30:29.960762 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:29.960648    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d77fa865-aa05-4f6b-bf94-fceea608785a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-775568978-sxkzc\" (UID: \"d77fa865-aa05-4f6b-bf94-fceea608785a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775568978-sxkzc"
Sep 17 18:30:29.960762 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:29.960710    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d77fa865-aa05-4f6b-bf94-fceea608785a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-775568978-sxkzc\" (UID: \"d77fa865-aa05-4f6b-bf94-fceea608785a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775568978-sxkzc"
Sep 17 18:30:29.960762 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:29.960736    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkjb7\" (UniqueName: \"kubernetes.io/projected/d77fa865-aa05-4f6b-bf94-fceea608785a-kube-api-access-lkjb7\") pod \"cluster-proxy-proxy-agent-775568978-sxkzc\" (UID: \"d77fa865-aa05-4f6b-bf94-fceea608785a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775568978-sxkzc"
Sep 17 18:30:29.960946 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:29.960820    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d77fa865-aa05-4f6b-bf94-fceea608785a-ca\") pod \"cluster-proxy-proxy-agent-775568978-sxkzc\" (UID: \"d77fa865-aa05-4f6b-bf94-fceea608785a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775568978-sxkzc"
Sep 17 18:30:29.960946 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:29.960837    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d77fa865-aa05-4f6b-bf94-fceea608785a-hub\") pod \"cluster-proxy-proxy-agent-775568978-sxkzc\" (UID: \"d77fa865-aa05-4f6b-bf94-fceea608785a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775568978-sxkzc"
Sep 17 18:30:29.960946 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:29.960861    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmrck\" (UniqueName: \"kubernetes.io/projected/14850890-1e36-4ab7-b480-516b4d81655e-kube-api-access-dmrck\") pod \"managed-serviceaccount-addon-agent-5b89b77694-4hs2h\" (UID: \"14850890-1e36-4ab7-b480-516b4d81655e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b89b77694-4hs2h"
Sep 17 18:30:30.041734 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:30.041705    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:30.061832 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:30.061805    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d77fa865-aa05-4f6b-bf94-fceea608785a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-775568978-sxkzc\" (UID: \"d77fa865-aa05-4f6b-bf94-fceea608785a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775568978-sxkzc"
Sep 17 18:30:30.061961 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:30.061840    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d77fa865-aa05-4f6b-bf94-fceea608785a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-775568978-sxkzc\" (UID: \"d77fa865-aa05-4f6b-bf94-fceea608785a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775568978-sxkzc"
Sep 17 18:30:30.061961 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:30.061866    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d77fa865-aa05-4f6b-bf94-fceea608785a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-775568978-sxkzc\" (UID: \"d77fa865-aa05-4f6b-bf94-fceea608785a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775568978-sxkzc"
Sep 17 18:30:30.061961 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:30.061883    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkjb7\" (UniqueName: \"kubernetes.io/projected/d77fa865-aa05-4f6b-bf94-fceea608785a-kube-api-access-lkjb7\") pod \"cluster-proxy-proxy-agent-775568978-sxkzc\" (UID: \"d77fa865-aa05-4f6b-bf94-fceea608785a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775568978-sxkzc"
Sep 17 18:30:30.061961 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:30.061941    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d77fa865-aa05-4f6b-bf94-fceea608785a-ca\") pod \"cluster-proxy-proxy-agent-775568978-sxkzc\" (UID: \"d77fa865-aa05-4f6b-bf94-fceea608785a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775568978-sxkzc"
Sep 17 18:30:30.061961 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:30.061957    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d77fa865-aa05-4f6b-bf94-fceea608785a-hub\") pod \"cluster-proxy-proxy-agent-775568978-sxkzc\" (UID: \"d77fa865-aa05-4f6b-bf94-fceea608785a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775568978-sxkzc"
Sep 17 18:30:30.062184 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:30.061976    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmrck\" (UniqueName: \"kubernetes.io/projected/14850890-1e36-4ab7-b480-516b4d81655e-kube-api-access-dmrck\") pod \"managed-serviceaccount-addon-agent-5b89b77694-4hs2h\" (UID: \"14850890-1e36-4ab7-b480-516b4d81655e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b89b77694-4hs2h"
Sep 17 18:30:30.062184 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:30.062157    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/14850890-1e36-4ab7-b480-516b4d81655e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5b89b77694-4hs2h\" (UID: \"14850890-1e36-4ab7-b480-516b4d81655e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b89b77694-4hs2h"
Sep 17 18:30:30.062700 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:30.062668    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d77fa865-aa05-4f6b-bf94-fceea608785a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-775568978-sxkzc\" (UID: \"d77fa865-aa05-4f6b-bf94-fceea608785a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775568978-sxkzc"
Sep 17 18:30:30.064816 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:30.064796    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d77fa865-aa05-4f6b-bf94-fceea608785a-ca\") pod \"cluster-proxy-proxy-agent-775568978-sxkzc\" (UID: \"d77fa865-aa05-4f6b-bf94-fceea608785a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775568978-sxkzc"
Sep 17 18:30:30.065018 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:30.064989    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d77fa865-aa05-4f6b-bf94-fceea608785a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-775568978-sxkzc\" (UID: \"d77fa865-aa05-4f6b-bf94-fceea608785a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775568978-sxkzc"
Sep 17 18:30:30.065018 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:30.065010    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/14850890-1e36-4ab7-b480-516b4d81655e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5b89b77694-4hs2h\" (UID: \"14850890-1e36-4ab7-b480-516b4d81655e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b89b77694-4hs2h"
Sep 17 18:30:30.065183 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:30.065068    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d77fa865-aa05-4f6b-bf94-fceea608785a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-775568978-sxkzc\" (UID: \"d77fa865-aa05-4f6b-bf94-fceea608785a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775568978-sxkzc"
Sep 17 18:30:30.065243 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:30.065210    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d77fa865-aa05-4f6b-bf94-fceea608785a-hub\") pod \"cluster-proxy-proxy-agent-775568978-sxkzc\" (UID: \"d77fa865-aa05-4f6b-bf94-fceea608785a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775568978-sxkzc"
Sep 17 18:30:30.081291 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:30.081265    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkjb7\" (UniqueName: \"kubernetes.io/projected/d77fa865-aa05-4f6b-bf94-fceea608785a-kube-api-access-lkjb7\") pod \"cluster-proxy-proxy-agent-775568978-sxkzc\" (UID: \"d77fa865-aa05-4f6b-bf94-fceea608785a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775568978-sxkzc"
Sep 17 18:30:30.081877 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:30.081855    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmrck\" (UniqueName: \"kubernetes.io/projected/14850890-1e36-4ab7-b480-516b4d81655e-kube-api-access-dmrck\") pod \"managed-serviceaccount-addon-agent-5b89b77694-4hs2h\" (UID: \"14850890-1e36-4ab7-b480-516b4d81655e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b89b77694-4hs2h"
Sep 17 18:30:30.159165 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:30.159069    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b89b77694-4hs2h"
Sep 17 18:30:30.164485 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:30.164464    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:30.185584 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:30.185555    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775568978-sxkzc"
Sep 17 18:30:30.278667 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:30.278633    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:30.384113 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:30.384081    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:30.437608 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:30.437514    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b89b77694-4hs2h"]
Sep 17 18:30:30.440408 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:30:30.440382    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14850890_1e36_4ab7_b480_516b4d81655e.slice/crio-4b9f437430e5465a67c5ae5931480c4674b6e8cb0dbf1033ddccba9fed66883e WatchSource:0}: Error finding container 4b9f437430e5465a67c5ae5931480c4674b6e8cb0dbf1033ddccba9fed66883e: Status 404 returned error can't find the container with id 4b9f437430e5465a67c5ae5931480c4674b6e8cb0dbf1033ddccba9fed66883e
Sep 17 18:30:30.475328 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:30.475302    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775568978-sxkzc"]
Sep 17 18:30:30.478386 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:30:30.478360    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd77fa865_aa05_4f6b_bf94_fceea608785a.slice/crio-bdb96fdfec220521b41134b94d161082ce9d792d23b271098112f24814b13037 WatchSource:0}: Error finding container bdb96fdfec220521b41134b94d161082ce9d792d23b271098112f24814b13037: Status 404 returned error can't find the container with id bdb96fdfec220521b41134b94d161082ce9d792d23b271098112f24814b13037
Sep 17 18:30:30.827980 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:30.827942    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775568978-sxkzc" event={"ID":"d77fa865-aa05-4f6b-bf94-fceea608785a","Type":"ContainerStarted","Data":"bdb96fdfec220521b41134b94d161082ce9d792d23b271098112f24814b13037"}
Sep 17 18:30:30.829009 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:30.828987    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b89b77694-4hs2h" event={"ID":"14850890-1e36-4ab7-b480-516b4d81655e","Type":"ContainerStarted","Data":"4b9f437430e5465a67c5ae5931480c4674b6e8cb0dbf1033ddccba9fed66883e"}
Sep 17 18:30:31.273761 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:31.273232    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c554cd64-2695-47d3-999a-2d8bf8d7ef63-original-pull-secret\") pod \"global-pull-secret-syncer-fs688\" (UID: \"c554cd64-2695-47d3-999a-2d8bf8d7ef63\") " pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:30:31.273761 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:31.273326    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c01ccd8-9e9c-467f-a2af-50720f272038-metrics-certs\") pod \"network-metrics-daemon-bnq8f\" (UID: \"1c01ccd8-9e9c-467f-a2af-50720f272038\") " pod="openshift-multus/network-metrics-daemon-bnq8f"
Sep 17 18:30:31.276224 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:31.276175    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 18:30:31.277256 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:31.277052    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 18:30:31.287212 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:31.287134    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c554cd64-2695-47d3-999a-2d8bf8d7ef63-original-pull-secret\") pod \"global-pull-secret-syncer-fs688\" (UID: \"c554cd64-2695-47d3-999a-2d8bf8d7ef63\") " pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:30:31.296853 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:31.296830    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c01ccd8-9e9c-467f-a2af-50720f272038-metrics-certs\") pod \"network-metrics-daemon-bnq8f\" (UID: \"1c01ccd8-9e9c-467f-a2af-50720f272038\") " pod="openshift-multus/network-metrics-daemon-bnq8f"
Sep 17 18:30:31.373779 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:31.373740    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcwlc\" (UniqueName: \"kubernetes.io/projected/9530dc1d-51a9-4c20-aa0f-dbfbf912e02e-kube-api-access-zcwlc\") pod \"network-check-target-cxndc\" (UID: \"9530dc1d-51a9-4c20-aa0f-dbfbf912e02e\") " pod="openshift-network-diagnostics/network-check-target-cxndc"
Sep 17 18:30:31.384710 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:31.384679    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcwlc\" (UniqueName: \"kubernetes.io/projected/9530dc1d-51a9-4c20-aa0f-dbfbf912e02e-kube-api-access-zcwlc\") pod \"network-check-target-cxndc\" (UID: \"9530dc1d-51a9-4c20-aa0f-dbfbf912e02e\") " pod="openshift-network-diagnostics/network-check-target-cxndc"
Sep 17 18:30:31.410510 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:31.410477    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fs688"
Sep 17 18:30:31.418103 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:31.418077    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-twvp2\""
Sep 17 18:30:31.425782 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:31.425763    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bnq8f"
Sep 17 18:30:31.426728 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:31.426706    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-58cdj\""
Sep 17 18:30:31.434131 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:31.434107    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cxndc"
Sep 17 18:30:31.474423 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:31.474390    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a7075b4-9ee1-4495-86e5-b2377d9b1b06-metrics-tls\") pod \"dns-default-jq7kv\" (UID: \"6a7075b4-9ee1-4495-86e5-b2377d9b1b06\") " pod="openshift-dns/dns-default-jq7kv"
Sep 17 18:30:31.477102 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:31.477075    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a7075b4-9ee1-4495-86e5-b2377d9b1b06-metrics-tls\") pod \"dns-default-jq7kv\" (UID: \"6a7075b4-9ee1-4495-86e5-b2377d9b1b06\") " pod="openshift-dns/dns-default-jq7kv"
Sep 17 18:30:31.708332 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:31.708104    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cvvrd\""
Sep 17 18:30:31.714615 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:31.714336    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jq7kv"
Sep 17 18:30:31.720618 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:31.720513    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fs688"]
Sep 17 18:30:31.745143 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:31.744195    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bnq8f"]
Sep 17 18:30:31.751629 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:31.750446    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cxndc"]
Sep 17 18:30:31.753314 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:30:31.752793    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c01ccd8_9e9c_467f_a2af_50720f272038.slice/crio-1731928382b135c56dfb0d58b5542cec9348603872a67fcac08b31e86bfda475 WatchSource:0}: Error finding container 1731928382b135c56dfb0d58b5542cec9348603872a67fcac08b31e86bfda475: Status 404 returned error can't find the container with id 1731928382b135c56dfb0d58b5542cec9348603872a67fcac08b31e86bfda475
Sep 17 18:30:31.753314 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:30:31.753237    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9530dc1d_51a9_4c20_aa0f_dbfbf912e02e.slice/crio-520bd51a380afb0461d5d66dea2f4b9010092cd2c28f5534d8c984dde498e343 WatchSource:0}: Error finding container 520bd51a380afb0461d5d66dea2f4b9010092cd2c28f5534d8c984dde498e343: Status 404 returned error can't find the container with id 520bd51a380afb0461d5d66dea2f4b9010092cd2c28f5534d8c984dde498e343
Sep 17 18:30:31.837652 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:31.837603    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fs688" event={"ID":"c554cd64-2695-47d3-999a-2d8bf8d7ef63","Type":"ContainerStarted","Data":"4ef6f9e1c56cb88c9a82517b97c05aa6c3c333e56fadb6e198157ca9ca83cf42"}
Sep 17 18:30:31.840343 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:31.840224    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cxndc" event={"ID":"9530dc1d-51a9-4c20-aa0f-dbfbf912e02e","Type":"ContainerStarted","Data":"520bd51a380afb0461d5d66dea2f4b9010092cd2c28f5534d8c984dde498e343"}
Sep 17 18:30:31.842204 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:31.842169    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bnq8f" event={"ID":"1c01ccd8-9e9c-467f-a2af-50720f272038","Type":"ContainerStarted","Data":"1731928382b135c56dfb0d58b5542cec9348603872a67fcac08b31e86bfda475"}
Sep 17 18:30:32.027325 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:32.026890    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jq7kv"]
Sep 17 18:30:32.809968 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:30:32.809929    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a7075b4_9ee1_4495_86e5_b2377d9b1b06.slice/crio-b4d70165d9352db483c56a007c800ca894d104bdfcd344add4f7062eb0457614 WatchSource:0}: Error finding container b4d70165d9352db483c56a007c800ca894d104bdfcd344add4f7062eb0457614: Status 404 returned error can't find the container with id b4d70165d9352db483c56a007c800ca894d104bdfcd344add4f7062eb0457614
Sep 17 18:30:32.848459 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:32.848192    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cxndc" event={"ID":"9530dc1d-51a9-4c20-aa0f-dbfbf912e02e","Type":"ContainerStarted","Data":"09c069c191c6af1013c22561c47f3f17da8212fdc3fb11108053bf02766a1a3e"}
Sep 17 18:30:32.848459 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:32.848306    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cxndc"
Sep 17 18:30:32.849650 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:32.849583    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jq7kv" event={"ID":"6a7075b4-9ee1-4495-86e5-b2377d9b1b06","Type":"ContainerStarted","Data":"b4d70165d9352db483c56a007c800ca894d104bdfcd344add4f7062eb0457614"}
Sep 17 18:30:32.872675 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:32.872623    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cxndc" podStartSLOduration=67.872608016 podStartE2EDuration="1m7.872608016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:32.870707693 +0000 UTC m=+69.721713093" watchObservedRunningTime="2026-09-17 18:30:32.872608016 +0000 UTC m=+69.723613414"
Sep 17 18:30:33.899896 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:33.899849    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/49a517a5-36f1-4482-b8fc-fa78fb902040-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-j5s56\" (UID: \"49a517a5-36f1-4482-b8fc-fa78fb902040\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-j5s56"
Sep 17 18:30:33.904463 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:33.904410    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/49a517a5-36f1-4482-b8fc-fa78fb902040-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-j5s56\" (UID: \"49a517a5-36f1-4482-b8fc-fa78fb902040\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-j5s56"
Sep 17 18:30:34.068511 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:34.068229    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-zcf4g\""
Sep 17 18:30:34.076781 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:34.076752    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-j5s56"
Sep 17 18:30:35.356004 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:35.355973    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:35.430801 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:35.430763    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:37.159485 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:37.159352    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-j5s56"]
Sep 17 18:30:37.164572 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:30:37.164400    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a517a5_36f1_4482_b8fc_fa78fb902040.slice/crio-99b6d4ed943edb9d0fafae74cb5147ea61ba91413042ddcaa8461ab593788fef WatchSource:0}: Error finding container 99b6d4ed943edb9d0fafae74cb5147ea61ba91413042ddcaa8461ab593788fef: Status 404 returned error can't find the container with id 99b6d4ed943edb9d0fafae74cb5147ea61ba91413042ddcaa8461ab593788fef
Sep 17 18:30:37.873932 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:37.873891    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bnq8f" event={"ID":"1c01ccd8-9e9c-467f-a2af-50720f272038","Type":"ContainerStarted","Data":"32bc7845955114b4b35633aa7a4520e48672f3a6fdbc1f5e56291980156aba7f"}
Sep 17 18:30:37.874139 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:37.873964    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bnq8f" event={"ID":"1c01ccd8-9e9c-467f-a2af-50720f272038","Type":"ContainerStarted","Data":"673602d661b1eade0b4c3ee88a594c8a0178010949614ea9273189630571a2fb"}
Sep 17 18:30:37.875267 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:37.875232    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-j5s56" event={"ID":"49a517a5-36f1-4482-b8fc-fa78fb902040","Type":"ContainerStarted","Data":"99b6d4ed943edb9d0fafae74cb5147ea61ba91413042ddcaa8461ab593788fef"}
Sep 17 18:30:37.877191 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:37.877131    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fs688" event={"ID":"c554cd64-2695-47d3-999a-2d8bf8d7ef63","Type":"ContainerStarted","Data":"428e3a76532fe8b1890f5671fb00d535b9ead2155b0a06d04b21d9d2426d78bd"}
Sep 17 18:30:37.879557 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:37.879454    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775568978-sxkzc" event={"ID":"d77fa865-aa05-4f6b-bf94-fceea608785a","Type":"ContainerStarted","Data":"640b0d2731fdda8ef538fa1a1d3e5f086ac09feeff5b23ec3950356909e3be55"}
Sep 17 18:30:37.879557 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:37.879495    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775568978-sxkzc" event={"ID":"d77fa865-aa05-4f6b-bf94-fceea608785a","Type":"ContainerStarted","Data":"4c971d905ebf432b570936c5cf5d231a20d67941790644bbc2d7101d997ec38e"}
Sep 17 18:30:37.879557 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:37.879511    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775568978-sxkzc" event={"ID":"d77fa865-aa05-4f6b-bf94-fceea608785a","Type":"ContainerStarted","Data":"f198ac2a47606dbd40608c5c7397a7d2cadc982cc079d977676e298a555eb81b"}
Sep 17 18:30:37.881268 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:37.881224    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b89b77694-4hs2h" event={"ID":"14850890-1e36-4ab7-b480-516b4d81655e","Type":"ContainerStarted","Data":"c78a494108c4c35d8b8de1b353de34ab55c6f7010bbd0a445940862a80406333"}
Sep 17 18:30:37.883238 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:37.883212    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jq7kv" event={"ID":"6a7075b4-9ee1-4495-86e5-b2377d9b1b06","Type":"ContainerStarted","Data":"0a04713783dc3fbdd6de9654059256e7498daf8afe2798ab6ee7c562837b5c14"}
Sep 17 18:30:37.883323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:37.883262    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jq7kv" event={"ID":"6a7075b4-9ee1-4495-86e5-b2377d9b1b06","Type":"ContainerStarted","Data":"f7ed6e4bb1c073486024e03044186875f897e1c24f87978b35918e38499fa286"}
Sep 17 18:30:37.883323 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:37.883305    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jq7kv"
Sep 17 18:30:37.891673 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:37.891623    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bnq8f" podStartSLOduration=67.773984206 podStartE2EDuration="1m12.891607091s" totalImagesPullingTime="5.117622885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:37.890798669 +0000 UTC m=+74.741804069" watchObservedRunningTime="2026-09-17 18:30:37.891607091 +0000 UTC m=+74.742612489"
Sep 17 18:30:37.908471 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:37.908423    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b89b77694-4hs2h" podStartSLOduration=2.505722681 podStartE2EDuration="8.908402826s" totalImagesPullingTime="6.402680145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:37.907106204 +0000 UTC m=+74.758111604" watchObservedRunningTime="2026-09-17 18:30:37.908402826 +0000 UTC m=+74.759408226"
Sep 17 18:30:37.927833 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:37.927768    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775568978-sxkzc" podStartSLOduration=2.5352107999999998 podStartE2EDuration="8.927749343s" totalImagesPullingTime="6.392538543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:37.927287997 +0000 UTC m=+74.778293397" watchObservedRunningTime="2026-09-17 18:30:37.927749343 +0000 UTC m=+74.778754741"
Sep 17 18:30:37.946627 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:37.946574    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fs688" podStartSLOduration=67.783187865 podStartE2EDuration="1m12.946557912s" totalImagesPullingTime="5.163370047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:37.94535955 +0000 UTC m=+74.796364950" watchObservedRunningTime="2026-09-17 18:30:37.946557912 +0000 UTC m=+74.797563311"
Sep 17 18:30:37.968625 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:37.968572    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jq7kv" podStartSLOduration=34.903007075 podStartE2EDuration="38.968550102s" totalImagesPullingTime="4.065543027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:37.966718635 +0000 UTC m=+74.817724047" watchObservedRunningTime="2026-09-17 18:30:37.968550102 +0000 UTC m=+74.819555497"
Sep 17 18:30:38.000722 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:38.000691    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:38.076722 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:38.076678    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:38.891258 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:38.891216    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-j5s56" event={"ID":"49a517a5-36f1-4482-b8fc-fa78fb902040","Type":"ContainerStarted","Data":"7833385be9eb69190804f5832fd2f7c6c8e27a084b6e9782379eb583d71d62e1"}
Sep 17 18:30:38.899177 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:38.899143    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:38.914104 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:38.914046    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-j5s56" podStartSLOduration=36.323943513 podStartE2EDuration="37.914025319s" totalImagesPullingTime="1.590081806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:38.91199887 +0000 UTC m=+75.763004268" watchObservedRunningTime="2026-09-17 18:30:38.914025319 +0000 UTC m=+75.765030719"
Sep 17 18:30:39.002906 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:39.002860    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:39.099597 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:39.099562    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:39.246057 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:39.246025    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:39.577619 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:39.577590    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:39.905951 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:39.905877    2588 patch_prober.go:28] interesting pod/image-registry-587f5c755b-xds6m container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 18:30:39.906301 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:39.905936    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-587f5c755b-xds6m" podUID="c6567c1f-021d-488b-8e11-8890cd9a8152" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:30:44.300875 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:44.300834    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:44.458156 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:44.458121    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:44.595506 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:44.595413    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:44.730024 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:44.729989    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:45.911685 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:45.911655    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:45.990770 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:45.990739    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:47.895169 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:47.895141    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jq7kv"
Sep 17 18:30:49.156496 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:49.156464    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:49.905399 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:49.905370    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:50.792366 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:50.792331    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:51.500475 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:51.500429    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:51.669194 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:51.669157    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:51.784405 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:51.784315    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:51.803544 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:51.803489    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cfccv"]
Sep 17 18:30:51.810805 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:51.810776    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cfccv"
Sep 17 18:30:51.813322 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:51.813298    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kwtnd\""
Sep 17 18:30:51.813756 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:51.813733    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 18:30:51.814076 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:51.814053    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 18:30:51.814416 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:51.814292    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 18:30:51.814522 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:51.814475    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 18:30:51.946543 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:51.946494    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/191b6fe8-f5c2-4a23-86b7-5b3f96adf26d-node-exporter-wtmp\") pod \"node-exporter-cfccv\" (UID: \"191b6fe8-f5c2-4a23-86b7-5b3f96adf26d\") " pod="openshift-monitoring/node-exporter-cfccv"
Sep 17 18:30:51.946717 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:51.946556    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/191b6fe8-f5c2-4a23-86b7-5b3f96adf26d-node-exporter-tls\") pod \"node-exporter-cfccv\" (UID: \"191b6fe8-f5c2-4a23-86b7-5b3f96adf26d\") " pod="openshift-monitoring/node-exporter-cfccv"
Sep 17 18:30:51.946717 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:51.946589    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/191b6fe8-f5c2-4a23-86b7-5b3f96adf26d-metrics-client-ca\") pod \"node-exporter-cfccv\" (UID: \"191b6fe8-f5c2-4a23-86b7-5b3f96adf26d\") " pod="openshift-monitoring/node-exporter-cfccv"
Sep 17 18:30:51.946717 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:51.946612    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/191b6fe8-f5c2-4a23-86b7-5b3f96adf26d-sys\") pod \"node-exporter-cfccv\" (UID: \"191b6fe8-f5c2-4a23-86b7-5b3f96adf26d\") " pod="openshift-monitoring/node-exporter-cfccv"
Sep 17 18:30:51.946717 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:51.946632    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/191b6fe8-f5c2-4a23-86b7-5b3f96adf26d-root\") pod \"node-exporter-cfccv\" (UID: \"191b6fe8-f5c2-4a23-86b7-5b3f96adf26d\") " pod="openshift-monitoring/node-exporter-cfccv"
Sep 17 18:30:51.946717 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:51.946654    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/191b6fe8-f5c2-4a23-86b7-5b3f96adf26d-node-exporter-textfile\") pod \"node-exporter-cfccv\" (UID: \"191b6fe8-f5c2-4a23-86b7-5b3f96adf26d\") " pod="openshift-monitoring/node-exporter-cfccv"
Sep 17 18:30:51.946717 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:51.946681    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/191b6fe8-f5c2-4a23-86b7-5b3f96adf26d-node-exporter-accelerators-collector-config\") pod \"node-exporter-cfccv\" (UID: \"191b6fe8-f5c2-4a23-86b7-5b3f96adf26d\") " pod="openshift-monitoring/node-exporter-cfccv"
Sep 17 18:30:51.947019 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:51.946752    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8xpv\" (UniqueName: \"kubernetes.io/projected/191b6fe8-f5c2-4a23-86b7-5b3f96adf26d-kube-api-access-j8xpv\") pod \"node-exporter-cfccv\" (UID: \"191b6fe8-f5c2-4a23-86b7-5b3f96adf26d\") " pod="openshift-monitoring/node-exporter-cfccv"
Sep 17 18:30:51.947019 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:51.946778    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/191b6fe8-f5c2-4a23-86b7-5b3f96adf26d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cfccv\" (UID: \"191b6fe8-f5c2-4a23-86b7-5b3f96adf26d\") " pod="openshift-monitoring/node-exporter-cfccv"
Sep 17 18:30:52.030747 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:52.030702    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:52.047837 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:52.047811    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8xpv\" (UniqueName: \"kubernetes.io/projected/191b6fe8-f5c2-4a23-86b7-5b3f96adf26d-kube-api-access-j8xpv\") pod \"node-exporter-cfccv\" (UID: \"191b6fe8-f5c2-4a23-86b7-5b3f96adf26d\") " pod="openshift-monitoring/node-exporter-cfccv"
Sep 17 18:30:52.047994 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:52.047847    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/191b6fe8-f5c2-4a23-86b7-5b3f96adf26d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cfccv\" (UID: \"191b6fe8-f5c2-4a23-86b7-5b3f96adf26d\") " pod="openshift-monitoring/node-exporter-cfccv"
Sep 17 18:30:52.047994 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:52.047885    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/191b6fe8-f5c2-4a23-86b7-5b3f96adf26d-node-exporter-wtmp\") pod \"node-exporter-cfccv\" (UID: \"191b6fe8-f5c2-4a23-86b7-5b3f96adf26d\") " pod="openshift-monitoring/node-exporter-cfccv"
Sep 17 18:30:52.047994 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:52.047910    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/191b6fe8-f5c2-4a23-86b7-5b3f96adf26d-node-exporter-tls\") pod \"node-exporter-cfccv\" (UID: \"191b6fe8-f5c2-4a23-86b7-5b3f96adf26d\") " pod="openshift-monitoring/node-exporter-cfccv"
Sep 17 18:30:52.047994 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:52.047940    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/191b6fe8-f5c2-4a23-86b7-5b3f96adf26d-metrics-client-ca\") pod \"node-exporter-cfccv\" (UID: \"191b6fe8-f5c2-4a23-86b7-5b3f96adf26d\") " pod="openshift-monitoring/node-exporter-cfccv"
Sep 17 18:30:52.047994 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:52.047965    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/191b6fe8-f5c2-4a23-86b7-5b3f96adf26d-sys\") pod \"node-exporter-cfccv\" (UID: \"191b6fe8-f5c2-4a23-86b7-5b3f96adf26d\") " pod="openshift-monitoring/node-exporter-cfccv"
Sep 17 18:30:52.047994 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:52.047984    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/191b6fe8-f5c2-4a23-86b7-5b3f96adf26d-root\") pod \"node-exporter-cfccv\" (UID: \"191b6fe8-f5c2-4a23-86b7-5b3f96adf26d\") " pod="openshift-monitoring/node-exporter-cfccv"
Sep 17 18:30:52.048267 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:52.048003    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/191b6fe8-f5c2-4a23-86b7-5b3f96adf26d-node-exporter-textfile\") pod \"node-exporter-cfccv\" (UID: \"191b6fe8-f5c2-4a23-86b7-5b3f96adf26d\") " pod="openshift-monitoring/node-exporter-cfccv"
Sep 17 18:30:52.048267 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:52.048033    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/191b6fe8-f5c2-4a23-86b7-5b3f96adf26d-node-exporter-accelerators-collector-config\") pod \"node-exporter-cfccv\" (UID: \"191b6fe8-f5c2-4a23-86b7-5b3f96adf26d\") " pod="openshift-monitoring/node-exporter-cfccv"
Sep 17 18:30:52.048267 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:52.048044    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/191b6fe8-f5c2-4a23-86b7-5b3f96adf26d-node-exporter-wtmp\") pod \"node-exporter-cfccv\" (UID: \"191b6fe8-f5c2-4a23-86b7-5b3f96adf26d\") " pod="openshift-monitoring/node-exporter-cfccv"
Sep 17 18:30:52.048267 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:52.048086    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/191b6fe8-f5c2-4a23-86b7-5b3f96adf26d-sys\") pod \"node-exporter-cfccv\" (UID: \"191b6fe8-f5c2-4a23-86b7-5b3f96adf26d\") " pod="openshift-monitoring/node-exporter-cfccv"
Sep 17 18:30:52.048267 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:52.048138    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/191b6fe8-f5c2-4a23-86b7-5b3f96adf26d-root\") pod \"node-exporter-cfccv\" (UID: \"191b6fe8-f5c2-4a23-86b7-5b3f96adf26d\") " pod="openshift-monitoring/node-exporter-cfccv"
Sep 17 18:30:52.048610 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:52.048576    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/191b6fe8-f5c2-4a23-86b7-5b3f96adf26d-node-exporter-textfile\") pod \"node-exporter-cfccv\" (UID: \"191b6fe8-f5c2-4a23-86b7-5b3f96adf26d\") " pod="openshift-monitoring/node-exporter-cfccv"
Sep 17 18:30:52.048906 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:52.048883    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/191b6fe8-f5c2-4a23-86b7-5b3f96adf26d-node-exporter-accelerators-collector-config\") pod \"node-exporter-cfccv\" (UID: \"191b6fe8-f5c2-4a23-86b7-5b3f96adf26d\") " pod="openshift-monitoring/node-exporter-cfccv"
Sep 17 18:30:52.048982 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:52.048914    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/191b6fe8-f5c2-4a23-86b7-5b3f96adf26d-metrics-client-ca\") pod \"node-exporter-cfccv\" (UID: \"191b6fe8-f5c2-4a23-86b7-5b3f96adf26d\") " pod="openshift-monitoring/node-exporter-cfccv"
Sep 17 18:30:52.050360 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:52.050329    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/191b6fe8-f5c2-4a23-86b7-5b3f96adf26d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cfccv\" (UID: \"191b6fe8-f5c2-4a23-86b7-5b3f96adf26d\") " pod="openshift-monitoring/node-exporter-cfccv"
Sep 17 18:30:52.050449 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:52.050403    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/191b6fe8-f5c2-4a23-86b7-5b3f96adf26d-node-exporter-tls\") pod \"node-exporter-cfccv\" (UID: \"191b6fe8-f5c2-4a23-86b7-5b3f96adf26d\") " pod="openshift-monitoring/node-exporter-cfccv"
Sep 17 18:30:52.060664 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:52.060641    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8xpv\" (UniqueName: \"kubernetes.io/projected/191b6fe8-f5c2-4a23-86b7-5b3f96adf26d-kube-api-access-j8xpv\") pod \"node-exporter-cfccv\" (UID: \"191b6fe8-f5c2-4a23-86b7-5b3f96adf26d\") " pod="openshift-monitoring/node-exporter-cfccv"
Sep 17 18:30:52.121648 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:52.121609    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cfccv"
Sep 17 18:30:52.132255 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:52.132231    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:52.134863 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:30:52.134829    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod191b6fe8_f5c2_4a23_86b7_5b3f96adf26d.slice/crio-c1027051577b53c35df7592e521337dfc6934b773c8366018c61e8056ffe97c9 WatchSource:0}: Error finding container c1027051577b53c35df7592e521337dfc6934b773c8366018c61e8056ffe97c9: Status 404 returned error can't find the container with id c1027051577b53c35df7592e521337dfc6934b773c8366018c61e8056ffe97c9
Sep 17 18:30:52.260883 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:52.260852    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:52.606851 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:52.606814    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:52.752717 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:52.752688    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:52.856743 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:52.856712    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:52.945796 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:52.945704    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cfccv" event={"ID":"191b6fe8-f5c2-4a23-86b7-5b3f96adf26d","Type":"ContainerStarted","Data":"c1027051577b53c35df7592e521337dfc6934b773c8366018c61e8056ffe97c9"}
Sep 17 18:30:52.962457 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:52.962423    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:53.461726 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:53.461691    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:53.567682 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:53.567651    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:53.783718 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:53.783680    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:53.873872 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:53.873787    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:53.951517 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:53.951482    2588 generic.go:356] "Generic (PLEG): container finished" podID="191b6fe8-f5c2-4a23-86b7-5b3f96adf26d" containerID="e2dbbc035a4153dc07fbba43ea3627b779aab3de179f0c35d11198101b48b678" exitCode=0
Sep 17 18:30:53.951700 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:53.951565    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cfccv" event={"ID":"191b6fe8-f5c2-4a23-86b7-5b3f96adf26d","Type":"ContainerDied","Data":"e2dbbc035a4153dc07fbba43ea3627b779aab3de179f0c35d11198101b48b678"}
Sep 17 18:30:54.478940 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:54.478888    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:54.757847 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:54.757765    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:54.856131 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:54.856102    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:54.919561 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:54.919507    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-587f5c755b-xds6m" podUID="c6567c1f-021d-488b-8e11-8890cd9a8152" containerName="registry" containerID="cri-o://61a422e2655612a306f5c793298e921237f506596b9ef0eb6fa5510c416ff19b" gracePeriod=30
Sep 17 18:30:54.957615 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:54.957579    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cfccv" event={"ID":"191b6fe8-f5c2-4a23-86b7-5b3f96adf26d","Type":"ContainerStarted","Data":"38eb5671805d713f37f151471a41cff241cb5f4f249d5dc6b48a4c9fa6e13ff6"}
Sep 17 18:30:54.957780 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:54.957641    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cfccv" event={"ID":"191b6fe8-f5c2-4a23-86b7-5b3f96adf26d","Type":"ContainerStarted","Data":"8ad77742f499240253975c061ae42c7399c2e364e43a7f2e450aa84925b28310"}
Sep 17 18:30:54.959657 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:54.959636    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:54.981144 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:54.981101    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cfccv" podStartSLOduration=2.774814903 podStartE2EDuration="3.981084497s" totalImagesPullingTime="1.145088954s" totalInitContainerRuntime="61.18064ms" isStatefulPod=true podCreationTimestamp="2026-09-17 18:30:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:54.979358865 +0000 UTC m=+91.830364269" watchObservedRunningTime="2026-09-17 18:30:54.981084497 +0000 UTC m=+91.832089896"
Sep 17 18:30:55.162460 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.162437    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:55.192588 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.192560    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:55.276870 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.276823    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6567c1f-021d-488b-8e11-8890cd9a8152-bound-sa-token\") pod \"c6567c1f-021d-488b-8e11-8890cd9a8152\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") "
Sep 17 18:30:55.276870 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.276866    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c6567c1f-021d-488b-8e11-8890cd9a8152-registry-certificates\") pod \"c6567c1f-021d-488b-8e11-8890cd9a8152\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") "
Sep 17 18:30:55.277118 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.276902    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6567c1f-021d-488b-8e11-8890cd9a8152-trusted-ca\") pod \"c6567c1f-021d-488b-8e11-8890cd9a8152\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") "
Sep 17 18:30:55.277118 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.276921    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlhh7\" (UniqueName: \"kubernetes.io/projected/c6567c1f-021d-488b-8e11-8890cd9a8152-kube-api-access-qlhh7\") pod \"c6567c1f-021d-488b-8e11-8890cd9a8152\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") "
Sep 17 18:30:55.277118 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.276942    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c6567c1f-021d-488b-8e11-8890cd9a8152-image-registry-private-configuration\") pod \"c6567c1f-021d-488b-8e11-8890cd9a8152\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") "
Sep 17 18:30:55.277118 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.276964    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6567c1f-021d-488b-8e11-8890cd9a8152-registry-tls\") pod \"c6567c1f-021d-488b-8e11-8890cd9a8152\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") "
Sep 17 18:30:55.277118 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.276993    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c6567c1f-021d-488b-8e11-8890cd9a8152-installation-pull-secrets\") pod \"c6567c1f-021d-488b-8e11-8890cd9a8152\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") "
Sep 17 18:30:55.277118 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.277022    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c6567c1f-021d-488b-8e11-8890cd9a8152-ca-trust-extracted\") pod \"c6567c1f-021d-488b-8e11-8890cd9a8152\" (UID: \"c6567c1f-021d-488b-8e11-8890cd9a8152\") "
Sep 17 18:30:55.277428 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.277402    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6567c1f-021d-488b-8e11-8890cd9a8152-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c6567c1f-021d-488b-8e11-8890cd9a8152" (UID: "c6567c1f-021d-488b-8e11-8890cd9a8152"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:30:55.277482 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.277412    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6567c1f-021d-488b-8e11-8890cd9a8152-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c6567c1f-021d-488b-8e11-8890cd9a8152" (UID: "c6567c1f-021d-488b-8e11-8890cd9a8152"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:30:55.279662 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.279619    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6567c1f-021d-488b-8e11-8890cd9a8152-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c6567c1f-021d-488b-8e11-8890cd9a8152" (UID: "c6567c1f-021d-488b-8e11-8890cd9a8152"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:30:55.279834 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.279810    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6567c1f-021d-488b-8e11-8890cd9a8152-kube-api-access-qlhh7" (OuterVolumeSpecName: "kube-api-access-qlhh7") pod "c6567c1f-021d-488b-8e11-8890cd9a8152" (UID: "c6567c1f-021d-488b-8e11-8890cd9a8152"). InnerVolumeSpecName "kube-api-access-qlhh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:30:55.279948 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.279866    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6567c1f-021d-488b-8e11-8890cd9a8152-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c6567c1f-021d-488b-8e11-8890cd9a8152" (UID: "c6567c1f-021d-488b-8e11-8890cd9a8152"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:30:55.279948 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.279883    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6567c1f-021d-488b-8e11-8890cd9a8152-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c6567c1f-021d-488b-8e11-8890cd9a8152" (UID: "c6567c1f-021d-488b-8e11-8890cd9a8152"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:30:55.279948 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.279919    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6567c1f-021d-488b-8e11-8890cd9a8152-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c6567c1f-021d-488b-8e11-8890cd9a8152" (UID: "c6567c1f-021d-488b-8e11-8890cd9a8152"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:30:55.285779 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.285757    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6567c1f-021d-488b-8e11-8890cd9a8152-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c6567c1f-021d-488b-8e11-8890cd9a8152" (UID: "c6567c1f-021d-488b-8e11-8890cd9a8152"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:55.378482 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.378452    2588 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6567c1f-021d-488b-8e11-8890cd9a8152-trusted-ca\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:30:55.378482 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.378478    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlhh7\" (UniqueName: \"kubernetes.io/projected/c6567c1f-021d-488b-8e11-8890cd9a8152-kube-api-access-qlhh7\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:30:55.378482 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.378489    2588 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c6567c1f-021d-488b-8e11-8890cd9a8152-image-registry-private-configuration\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:30:55.378731 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.378499    2588 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c6567c1f-021d-488b-8e11-8890cd9a8152-registry-tls\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:30:55.378731 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.378507    2588 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c6567c1f-021d-488b-8e11-8890cd9a8152-installation-pull-secrets\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:30:55.378731 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.378516    2588 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c6567c1f-021d-488b-8e11-8890cd9a8152-ca-trust-extracted\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:30:55.378731 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.378542    2588 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6567c1f-021d-488b-8e11-8890cd9a8152-bound-sa-token\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:30:55.378731 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.378551    2588 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c6567c1f-021d-488b-8e11-8890cd9a8152-registry-certificates\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:30:55.726162 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.726058    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:55.830975 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.830941    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:55.928704 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.928665    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:55.965159 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.965128    2588 generic.go:356] "Generic (PLEG): container finished" podID="c6567c1f-021d-488b-8e11-8890cd9a8152" containerID="61a422e2655612a306f5c793298e921237f506596b9ef0eb6fa5510c416ff19b" exitCode=0
Sep 17 18:30:55.965320 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.965195    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-587f5c755b-xds6m"
Sep 17 18:30:55.965320 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.965212    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-587f5c755b-xds6m" event={"ID":"c6567c1f-021d-488b-8e11-8890cd9a8152","Type":"ContainerDied","Data":"61a422e2655612a306f5c793298e921237f506596b9ef0eb6fa5510c416ff19b"}
Sep 17 18:30:55.965320 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.965296    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-587f5c755b-xds6m" event={"ID":"c6567c1f-021d-488b-8e11-8890cd9a8152","Type":"ContainerDied","Data":"ae17008b3c49e1460f52025fcc8116694b41338644df7cca4f69c2710b3d15da"}
Sep 17 18:30:55.965444 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.965322    2588 scope.go:117] "RemoveContainer" containerID="61a422e2655612a306f5c793298e921237f506596b9ef0eb6fa5510c416ff19b"
Sep 17 18:30:55.973990 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.973970    2588 scope.go:117] "RemoveContainer" containerID="61a422e2655612a306f5c793298e921237f506596b9ef0eb6fa5510c416ff19b"
Sep 17 18:30:55.974311 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:30:55.974290    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61a422e2655612a306f5c793298e921237f506596b9ef0eb6fa5510c416ff19b\": container with ID starting with 61a422e2655612a306f5c793298e921237f506596b9ef0eb6fa5510c416ff19b not found: ID does not exist" containerID="61a422e2655612a306f5c793298e921237f506596b9ef0eb6fa5510c416ff19b"
Sep 17 18:30:55.974365 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.974319    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61a422e2655612a306f5c793298e921237f506596b9ef0eb6fa5510c416ff19b"} err="failed to get container status \"61a422e2655612a306f5c793298e921237f506596b9ef0eb6fa5510c416ff19b\": rpc error: code = NotFound desc = could not find container \"61a422e2655612a306f5c793298e921237f506596b9ef0eb6fa5510c416ff19b\": container with ID starting with 61a422e2655612a306f5c793298e921237f506596b9ef0eb6fa5510c416ff19b not found: ID does not exist"
Sep 17 18:30:55.984371 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.984298    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-587f5c755b-xds6m"]
Sep 17 18:30:55.988266 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:55.988217    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-587f5c755b-xds6m"]
Sep 17 18:30:56.513504 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:56.513466    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:56.619442 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:56.619412    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:56.710232 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:56.710198    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:57.504984 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:57.504951    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6567c1f-021d-488b-8e11-8890cd9a8152" path="/var/lib/kubelet/pods/c6567c1f-021d-488b-8e11-8890cd9a8152/volumes"
Sep 17 18:30:57.642739 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:57.642707    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:57.704108 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:57.704080    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:58.119813 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:30:58.119770    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:00.354459 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:31:00.354425    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:00.445950 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:31:00.445912    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:03.760327 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:31:03.760293    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:03.822255 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:31:03.822196    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:03.857658 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:31:03.857629    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cxndc"
Sep 17 18:31:06.209736 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:31:06.209700    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:06.311110 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:31:06.311075    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:08.734393 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:31:08.734360    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:08.843348 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:31:08.843321    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:08.945366 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:31:08.945338    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:12.527179 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:31:12.527146    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:12.631518 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:31:12.631486    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:12.724018 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:31:12.723980    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:12.834064 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:31:12.834030    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:12.943310 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:31:12.943279    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:15.440344 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:31:15.440311    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:15.541640 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:31:15.541588    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:27.640655 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:31:27.640609    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:27.705365 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:31:27.705334    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:33.107344 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:31:33.107309    2588 generic.go:356] "Generic (PLEG): container finished" podID="63716ecc-bcb1-44d0-96c4-40ac49599a79" containerID="2f62a541a0f59ce58ab499cce5014c9ca5ef0078f531412a87ed78059e60a812" exitCode=0
Sep 17 18:31:33.107847 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:31:33.107382    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-cf8bw" event={"ID":"63716ecc-bcb1-44d0-96c4-40ac49599a79","Type":"ContainerDied","Data":"2f62a541a0f59ce58ab499cce5014c9ca5ef0078f531412a87ed78059e60a812"}
Sep 17 18:31:33.107847 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:31:33.107741    2588 scope.go:117] "RemoveContainer" containerID="2f62a541a0f59ce58ab499cce5014c9ca5ef0078f531412a87ed78059e60a812"
Sep 17 18:31:34.045893 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:31:34.045862    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6llb8_676d0454-70c3-4491-b9bc-7d934002b6be/serve-healthcheck-canary/0.log"
Sep 17 18:31:34.113590 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:31:34.113551    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-cf8bw" event={"ID":"63716ecc-bcb1-44d0-96c4-40ac49599a79","Type":"ContainerStarted","Data":"e947219ba47544c919e38c4b9f38c40ec4128c5cdb572259467c50ba47d98b16"}
Sep 17 18:31:37.201437 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:31:37.201404    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:37.384273 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:31:37.384241    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:37.482845 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:31:37.482758    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:58.121977 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:31:58.121922    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:04.698779 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:32:04.698745    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:11.060159 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:32:11.060082    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:11.525814 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:32:11.525730    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:20.770498 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:32:20.770458    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:20.891144 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:32:20.891108    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:26.388724 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:32:26.388692    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:26.502594 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:32:26.502561    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:38.536043 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:32:38.536006    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:38.665050 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:32:38.665022    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:41.208830 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:32:41.208790    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:41.351431 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:32:41.351397    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:42.859777 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:32:42.859742    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:43.057890 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:32:43.057856    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:04.158219 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:33:04.158184    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:04.238596 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:33:04.238567    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:24.037507 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:33:24.037473    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:24.159296 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:33:24.159265    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:31.157436 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:33:31.157401    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:37.078214 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:33:37.078140    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:54.067732 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:33:54.067683    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:54.167693 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:33:54.167660    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:00.383823 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:34:00.383791    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:00.768708 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:34:00.768627    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:00.988260 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:34:00.988224    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:01.153430 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:34:01.153396    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:01.329269 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:34:01.329218    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:01.433318 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:34:01.433235    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:01.719158 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:34:01.719073    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:01.908072 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:34:01.908037    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:02.110672 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:34:02.110635    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:02.206338 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:34:02.206306    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:25.439848 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:34:25.439811    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xlklb_07ee31e0-43bb-4940-837d-969bd840492b/console-operator/1.log"
Sep 17 18:34:25.445092 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:34:25.445067    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovn-acl-logging/0.log"
Sep 17 18:34:25.446543 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:34:25.446503    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xlklb_07ee31e0-43bb-4940-837d-969bd840492b/console-operator/1.log"
Sep 17 18:34:25.450447 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:34:25.450430    2588 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 18:34:25.451678 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:34:25.451661    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovn-acl-logging/0.log"
Sep 17 18:34:56.087703 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:34:56.087663    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:57.732040 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:34:57.732008    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:07.768004 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:07.767925    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:07.897966 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:07.897934    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:11.939158 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:11.939125    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:12.116334 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:12.116301    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:16.805460 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:16.805423    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:16.867857 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:16.867831    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:23.784789 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:23.784753    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:23.929975 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:23.929941    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:26.164969 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:26.164935    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:26.373863 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:26.373830    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:27.998731 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:27.998691    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:28.101232 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:28.101197    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:28.200205 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:28.200175    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:31.139416 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:31.139374    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-9xfnn"]
Sep 17 18:35:31.139949 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:31.139810    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6567c1f-021d-488b-8e11-8890cd9a8152" containerName="registry"
Sep 17 18:35:31.139949 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:31.139829    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6567c1f-021d-488b-8e11-8890cd9a8152" containerName="registry"
Sep 17 18:35:31.139949 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:31.139890    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6567c1f-021d-488b-8e11-8890cd9a8152" containerName="registry"
Sep 17 18:35:31.141806 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:31.141784    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-9xfnn"
Sep 17 18:35:31.144139 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:31.144116    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 17 18:35:31.144466 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:31.144446    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 17 18:35:31.145068 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:31.145051    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-m5sk6\""
Sep 17 18:35:31.155404 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:31.155378    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-9xfnn"]
Sep 17 18:35:31.268903 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:31.268868    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/579e2cb2-5db2-4be1-a37a-9bdad891ca79-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-9xfnn\" (UID: \"579e2cb2-5db2-4be1-a37a-9bdad891ca79\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9xfnn"
Sep 17 18:35:31.269069 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:31.268919    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wmjs\" (UniqueName: \"kubernetes.io/projected/579e2cb2-5db2-4be1-a37a-9bdad891ca79-kube-api-access-9wmjs\") pod \"cert-manager-webhook-6486ffd997-9xfnn\" (UID: \"579e2cb2-5db2-4be1-a37a-9bdad891ca79\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9xfnn"
Sep 17 18:35:31.369843 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:31.369806    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/579e2cb2-5db2-4be1-a37a-9bdad891ca79-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-9xfnn\" (UID: \"579e2cb2-5db2-4be1-a37a-9bdad891ca79\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9xfnn"
Sep 17 18:35:31.369983 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:31.369859    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wmjs\" (UniqueName: \"kubernetes.io/projected/579e2cb2-5db2-4be1-a37a-9bdad891ca79-kube-api-access-9wmjs\") pod \"cert-manager-webhook-6486ffd997-9xfnn\" (UID: \"579e2cb2-5db2-4be1-a37a-9bdad891ca79\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9xfnn"
Sep 17 18:35:31.379761 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:31.379735    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/579e2cb2-5db2-4be1-a37a-9bdad891ca79-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-9xfnn\" (UID: \"579e2cb2-5db2-4be1-a37a-9bdad891ca79\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9xfnn"
Sep 17 18:35:31.379894 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:31.379817    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wmjs\" (UniqueName: \"kubernetes.io/projected/579e2cb2-5db2-4be1-a37a-9bdad891ca79-kube-api-access-9wmjs\") pod \"cert-manager-webhook-6486ffd997-9xfnn\" (UID: \"579e2cb2-5db2-4be1-a37a-9bdad891ca79\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9xfnn"
Sep 17 18:35:31.450920 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:31.450839    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-9xfnn"
Sep 17 18:35:31.721465 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:31.721388    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-9xfnn"]
Sep 17 18:35:31.724650 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:35:31.724620    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod579e2cb2_5db2_4be1_a37a_9bdad891ca79.slice/crio-14e666d7243ac79eef1178c6802b6d1a88bfda53168d90c0f6115c4ce3f3cfdc WatchSource:0}: Error finding container 14e666d7243ac79eef1178c6802b6d1a88bfda53168d90c0f6115c4ce3f3cfdc: Status 404 returned error can't find the container with id 14e666d7243ac79eef1178c6802b6d1a88bfda53168d90c0f6115c4ce3f3cfdc
Sep 17 18:35:31.726703 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:31.726684    2588 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:35:31.958803 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:31.958768    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-9xfnn" event={"ID":"579e2cb2-5db2-4be1-a37a-9bdad891ca79","Type":"ContainerStarted","Data":"14e666d7243ac79eef1178c6802b6d1a88bfda53168d90c0f6115c4ce3f3cfdc"}
Sep 17 18:35:33.653184 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:33.653149    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:34.868734 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:34.868647    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:34.930742 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:34.930713    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:34.973100 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:34.973069    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-9xfnn" event={"ID":"579e2cb2-5db2-4be1-a37a-9bdad891ca79","Type":"ContainerStarted","Data":"bfad7e364ac19b6523a30ec7f0fb7a8e9da5c1eebade64a353fd9bb1adf4682c"}
Sep 17 18:35:34.973239 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:34.973121    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-9xfnn"
Sep 17 18:35:34.990542 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:34.990477    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-9xfnn" podStartSLOduration=1.097321525 podStartE2EDuration="3.990461208s" totalImagesPullingTime="2.893139683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:35:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:35:34.98914958 +0000 UTC m=+371.840154978" watchObservedRunningTime="2026-09-17 18:35:34.990461208 +0000 UTC m=+371.841466603"
Sep 17 18:35:39.298482 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:39.298443    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:39.366101 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:39.366075    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:40.673613 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:40.673566    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:40.773884 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:40.773851    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:40.979218 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:40.979140    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-9xfnn"
Sep 17 18:35:41.617022 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:41.616977    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:42.508008 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:42.507976    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vsrhx"]
Sep 17 18:35:42.510819 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:42.510803    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vsrhx"
Sep 17 18:35:42.513156 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:42.513132    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:35:42.513397 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:42.513381    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-pn2v7\""
Sep 17 18:35:42.514187 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:42.514172    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 18:35:42.519802 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:42.519781    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vsrhx"]
Sep 17 18:35:42.552824 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:42.552790    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxxr4\" (UniqueName: \"kubernetes.io/projected/85f3da13-01f5-47dc-a03d-7529532edff2-kube-api-access-wxxr4\") pod \"openshift-lws-operator-fd8ccff4c-vsrhx\" (UID: \"85f3da13-01f5-47dc-a03d-7529532edff2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vsrhx"
Sep 17 18:35:42.552968 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:42.552836    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85f3da13-01f5-47dc-a03d-7529532edff2-tmp\") pod \"openshift-lws-operator-fd8ccff4c-vsrhx\" (UID: \"85f3da13-01f5-47dc-a03d-7529532edff2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vsrhx"
Sep 17 18:35:42.653702 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:42.653668    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxxr4\" (UniqueName: \"kubernetes.io/projected/85f3da13-01f5-47dc-a03d-7529532edff2-kube-api-access-wxxr4\") pod \"openshift-lws-operator-fd8ccff4c-vsrhx\" (UID: \"85f3da13-01f5-47dc-a03d-7529532edff2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vsrhx"
Sep 17 18:35:42.653849 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:42.653726    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85f3da13-01f5-47dc-a03d-7529532edff2-tmp\") pod \"openshift-lws-operator-fd8ccff4c-vsrhx\" (UID: \"85f3da13-01f5-47dc-a03d-7529532edff2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vsrhx"
Sep 17 18:35:42.654134 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:42.654115    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85f3da13-01f5-47dc-a03d-7529532edff2-tmp\") pod \"openshift-lws-operator-fd8ccff4c-vsrhx\" (UID: \"85f3da13-01f5-47dc-a03d-7529532edff2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vsrhx"
Sep 17 18:35:42.666095 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:42.666072    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxxr4\" (UniqueName: \"kubernetes.io/projected/85f3da13-01f5-47dc-a03d-7529532edff2-kube-api-access-wxxr4\") pod \"openshift-lws-operator-fd8ccff4c-vsrhx\" (UID: \"85f3da13-01f5-47dc-a03d-7529532edff2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vsrhx"
Sep 17 18:35:42.820131 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:42.820098    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vsrhx"
Sep 17 18:35:43.089776 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:43.089689    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vsrhx"]
Sep 17 18:35:43.092581 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:35:43.092553    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85f3da13_01f5_47dc_a03d_7529532edff2.slice/crio-2df658c14954a14505573c4c40aab33d8ec48b23c09f5f9c92e202cad09c13a8 WatchSource:0}: Error finding container 2df658c14954a14505573c4c40aab33d8ec48b23c09f5f9c92e202cad09c13a8: Status 404 returned error can't find the container with id 2df658c14954a14505573c4c40aab33d8ec48b23c09f5f9c92e202cad09c13a8
Sep 17 18:35:44.008351 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:44.008301    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vsrhx" event={"ID":"85f3da13-01f5-47dc-a03d-7529532edff2","Type":"ContainerStarted","Data":"2df658c14954a14505573c4c40aab33d8ec48b23c09f5f9c92e202cad09c13a8"}
Sep 17 18:35:45.722969 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:45.722886    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:45.784280 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:45.784229    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:46.017485 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:46.017405    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vsrhx" event={"ID":"85f3da13-01f5-47dc-a03d-7529532edff2","Type":"ContainerStarted","Data":"9a7dc6cb0c4dc070bee2bb0877ac5d3433776be18d011ac35e352019e8d3cdac"}
Sep 17 18:35:46.035254 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:46.035205    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vsrhx" podStartSLOduration=1.674544619 podStartE2EDuration="4.035188981s" totalImagesPullingTime="2.360644362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:35:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:35:46.034272023 +0000 UTC m=+382.885277421" watchObservedRunningTime="2026-09-17 18:35:46.035188981 +0000 UTC m=+382.886194380"
Sep 17 18:35:51.481242 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:51.481205    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:52.850180 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:52.850150    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:52.913658 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:52.913625    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:52.983599 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:52.983569    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:56.457821 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:56.457789    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:56.524579 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:56.524542    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:57.716473 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:35:57.716439    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:00.253896 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:00.253862    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:00.360407 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:00.360376    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:02.934009 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:02.933977    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"]
Sep 17 18:36:02.940073 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:02.940041    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:36:02.942855 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:02.942826    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 18:36:02.943004 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:02.942873    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-njhkk\""
Sep 17 18:36:02.943004 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:02.942873    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 18:36:02.943330 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:02.943317    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 18:36:02.952549 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:02.952499    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"]
Sep 17 18:36:02.978774 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:02.978743    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:03.013515 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:03.013478    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/585c7370-1cb3-4035-ac56-be5c3f566027-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-72h4k\" (UID: \"585c7370-1cb3-4035-ac56-be5c3f566027\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:36:03.013668 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:03.013517    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/585c7370-1cb3-4035-ac56-be5c3f566027-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-72h4k\" (UID: \"585c7370-1cb3-4035-ac56-be5c3f566027\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:36:03.013668 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:03.013601    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldwjl\" (UniqueName: \"kubernetes.io/projected/585c7370-1cb3-4035-ac56-be5c3f566027-kube-api-access-ldwjl\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-72h4k\" (UID: \"585c7370-1cb3-4035-ac56-be5c3f566027\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:36:03.013745 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:03.013672    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/585c7370-1cb3-4035-ac56-be5c3f566027-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-72h4k\" (UID: \"585c7370-1cb3-4035-ac56-be5c3f566027\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:36:03.013745 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:03.013724    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/585c7370-1cb3-4035-ac56-be5c3f566027-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-72h4k\" (UID: \"585c7370-1cb3-4035-ac56-be5c3f566027\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:36:03.013823 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:03.013746    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/585c7370-1cb3-4035-ac56-be5c3f566027-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-72h4k\" (UID: \"585c7370-1cb3-4035-ac56-be5c3f566027\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:36:03.013823 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:03.013769    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/585c7370-1cb3-4035-ac56-be5c3f566027-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-72h4k\" (UID: \"585c7370-1cb3-4035-ac56-be5c3f566027\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:36:03.013823 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:03.013790    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/585c7370-1cb3-4035-ac56-be5c3f566027-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-72h4k\" (UID: \"585c7370-1cb3-4035-ac56-be5c3f566027\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:36:03.013918 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:03.013851    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/585c7370-1cb3-4035-ac56-be5c3f566027-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-72h4k\" (UID: \"585c7370-1cb3-4035-ac56-be5c3f566027\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:36:03.065390 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:03.065360    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:03.114987 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:03.114945    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/585c7370-1cb3-4035-ac56-be5c3f566027-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-72h4k\" (UID: \"585c7370-1cb3-4035-ac56-be5c3f566027\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:36:03.114987 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:03.114988    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/585c7370-1cb3-4035-ac56-be5c3f566027-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-72h4k\" (UID: \"585c7370-1cb3-4035-ac56-be5c3f566027\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:36:03.115215 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:03.115008    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/585c7370-1cb3-4035-ac56-be5c3f566027-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-72h4k\" (UID: \"585c7370-1cb3-4035-ac56-be5c3f566027\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:36:03.115215 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:03.115041    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldwjl\" (UniqueName: \"kubernetes.io/projected/585c7370-1cb3-4035-ac56-be5c3f566027-kube-api-access-ldwjl\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-72h4k\" (UID: \"585c7370-1cb3-4035-ac56-be5c3f566027\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:36:03.115215 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:03.115079    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/585c7370-1cb3-4035-ac56-be5c3f566027-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-72h4k\" (UID: \"585c7370-1cb3-4035-ac56-be5c3f566027\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:36:03.115215 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:03.115125    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/585c7370-1cb3-4035-ac56-be5c3f566027-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-72h4k\" (UID: \"585c7370-1cb3-4035-ac56-be5c3f566027\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:36:03.115215 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:03.115150    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/585c7370-1cb3-4035-ac56-be5c3f566027-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-72h4k\" (UID: \"585c7370-1cb3-4035-ac56-be5c3f566027\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:36:03.115215 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:03.115178    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/585c7370-1cb3-4035-ac56-be5c3f566027-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-72h4k\" (UID: \"585c7370-1cb3-4035-ac56-be5c3f566027\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:36:03.115215 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:03.115202    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/585c7370-1cb3-4035-ac56-be5c3f566027-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-72h4k\" (UID: \"585c7370-1cb3-4035-ac56-be5c3f566027\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:36:03.115669 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:03.115446    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/585c7370-1cb3-4035-ac56-be5c3f566027-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-72h4k\" (UID: \"585c7370-1cb3-4035-ac56-be5c3f566027\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:36:03.115669 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:03.115563    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/585c7370-1cb3-4035-ac56-be5c3f566027-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-72h4k\" (UID: \"585c7370-1cb3-4035-ac56-be5c3f566027\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:36:03.115777 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:03.115684    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/585c7370-1cb3-4035-ac56-be5c3f566027-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-72h4k\" (UID: \"585c7370-1cb3-4035-ac56-be5c3f566027\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:36:03.115890 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:03.115865    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/585c7370-1cb3-4035-ac56-be5c3f566027-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-72h4k\" (UID: \"585c7370-1cb3-4035-ac56-be5c3f566027\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:36:03.116026 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:03.115998    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/585c7370-1cb3-4035-ac56-be5c3f566027-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-72h4k\" (UID: \"585c7370-1cb3-4035-ac56-be5c3f566027\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:36:03.117428 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:03.117406    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/585c7370-1cb3-4035-ac56-be5c3f566027-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-72h4k\" (UID: \"585c7370-1cb3-4035-ac56-be5c3f566027\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:36:03.117655 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:03.117639    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/585c7370-1cb3-4035-ac56-be5c3f566027-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-72h4k\" (UID: \"585c7370-1cb3-4035-ac56-be5c3f566027\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:36:03.123423 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:03.123400    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/585c7370-1cb3-4035-ac56-be5c3f566027-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-72h4k\" (UID: \"585c7370-1cb3-4035-ac56-be5c3f566027\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:36:03.123773 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:03.123752    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldwjl\" (UniqueName: \"kubernetes.io/projected/585c7370-1cb3-4035-ac56-be5c3f566027-kube-api-access-ldwjl\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-72h4k\" (UID: \"585c7370-1cb3-4035-ac56-be5c3f566027\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:36:03.256850 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:03.256758    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:36:03.537732 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:03.537651    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"]
Sep 17 18:36:03.541188 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:36:03.541161    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod585c7370_1cb3_4035_ac56_be5c3f566027.slice/crio-3d5c6ed8650e23d1a0905d4b13438f16aa75633c145e4d4a9ac84a53133f7658 WatchSource:0}: Error finding container 3d5c6ed8650e23d1a0905d4b13438f16aa75633c145e4d4a9ac84a53133f7658: Status 404 returned error can't find the container with id 3d5c6ed8650e23d1a0905d4b13438f16aa75633c145e4d4a9ac84a53133f7658
Sep 17 18:36:04.082507 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:04.082470    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k" event={"ID":"585c7370-1cb3-4035-ac56-be5c3f566027","Type":"ContainerStarted","Data":"3d5c6ed8650e23d1a0905d4b13438f16aa75633c145e4d4a9ac84a53133f7658"}
Sep 17 18:36:06.207819 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:06.207786    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:06.615651 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:06.615612    2588 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 18:36:06.615734 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:06.615696    2588 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 18:36:06.615773 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:06.615738    2588 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 18:36:07.096755 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:07.096718    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k" event={"ID":"585c7370-1cb3-4035-ac56-be5c3f566027","Type":"ContainerStarted","Data":"49b7be038103af0e31313793fbfa9a34dbb46034e2c0a684030d8e1cb02f078e"}
Sep 17 18:36:07.123240 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:07.123191    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k" podStartSLOduration=2.051194977 podStartE2EDuration="5.123174235s" totalImagesPullingTime="3.071979258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:36:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:36:07.122106935 +0000 UTC m=+403.973112335" watchObservedRunningTime="2026-09-17 18:36:07.123174235 +0000 UTC m=+403.974179634"
Sep 17 18:36:07.257844 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:07.257811    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:36:07.261765 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:07.261745    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:36:08.100220 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:08.100193    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:36:08.101050 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:08.101028    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:36:08.599840 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:08.599811    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:08.904796 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:08.904689    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:12.248387 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:12.248356    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:12.323652 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:12.323620    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7594644fcd-x58df"]
Sep 17 18:36:12.328998 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:12.328978    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7594644fcd-x58df"
Sep 17 18:36:12.332842 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:12.332816    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 17 18:36:12.333082 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:12.333063    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 17 18:36:12.333146 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:12.333078    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-jhq2t\""
Sep 17 18:36:12.333146 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:12.333114    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 17 18:36:12.341332 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:12.341309    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7594644fcd-x58df"]
Sep 17 18:36:12.363847 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:12.363779    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:12.389149 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:12.389128    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfp6x\" (UniqueName: \"kubernetes.io/projected/f72915b1-27d1-41fe-94f6-eac990c3e860-kube-api-access-gfp6x\") pod \"lws-controller-manager-7594644fcd-x58df\" (UID: \"f72915b1-27d1-41fe-94f6-eac990c3e860\") " pod="openshift-lws-operator/lws-controller-manager-7594644fcd-x58df"
Sep 17 18:36:12.389262 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:12.389165    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f72915b1-27d1-41fe-94f6-eac990c3e860-manager-config\") pod \"lws-controller-manager-7594644fcd-x58df\" (UID: \"f72915b1-27d1-41fe-94f6-eac990c3e860\") " pod="openshift-lws-operator/lws-controller-manager-7594644fcd-x58df"
Sep 17 18:36:12.389262 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:12.389200    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f72915b1-27d1-41fe-94f6-eac990c3e860-metrics-cert\") pod \"lws-controller-manager-7594644fcd-x58df\" (UID: \"f72915b1-27d1-41fe-94f6-eac990c3e860\") " pod="openshift-lws-operator/lws-controller-manager-7594644fcd-x58df"
Sep 17 18:36:12.389355 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:12.389267    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f72915b1-27d1-41fe-94f6-eac990c3e860-cert\") pod \"lws-controller-manager-7594644fcd-x58df\" (UID: \"f72915b1-27d1-41fe-94f6-eac990c3e860\") " pod="openshift-lws-operator/lws-controller-manager-7594644fcd-x58df"
Sep 17 18:36:12.489919 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:12.489878    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f72915b1-27d1-41fe-94f6-eac990c3e860-metrics-cert\") pod \"lws-controller-manager-7594644fcd-x58df\" (UID: \"f72915b1-27d1-41fe-94f6-eac990c3e860\") " pod="openshift-lws-operator/lws-controller-manager-7594644fcd-x58df"
Sep 17 18:36:12.490105 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:12.489934    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f72915b1-27d1-41fe-94f6-eac990c3e860-cert\") pod \"lws-controller-manager-7594644fcd-x58df\" (UID: \"f72915b1-27d1-41fe-94f6-eac990c3e860\") " pod="openshift-lws-operator/lws-controller-manager-7594644fcd-x58df"
Sep 17 18:36:12.490105 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:12.489977    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfp6x\" (UniqueName: \"kubernetes.io/projected/f72915b1-27d1-41fe-94f6-eac990c3e860-kube-api-access-gfp6x\") pod \"lws-controller-manager-7594644fcd-x58df\" (UID: \"f72915b1-27d1-41fe-94f6-eac990c3e860\") " pod="openshift-lws-operator/lws-controller-manager-7594644fcd-x58df"
Sep 17 18:36:12.490105 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:12.490005    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f72915b1-27d1-41fe-94f6-eac990c3e860-manager-config\") pod \"lws-controller-manager-7594644fcd-x58df\" (UID: \"f72915b1-27d1-41fe-94f6-eac990c3e860\") " pod="openshift-lws-operator/lws-controller-manager-7594644fcd-x58df"
Sep 17 18:36:12.490727 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:12.490698    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f72915b1-27d1-41fe-94f6-eac990c3e860-manager-config\") pod \"lws-controller-manager-7594644fcd-x58df\" (UID: \"f72915b1-27d1-41fe-94f6-eac990c3e860\") " pod="openshift-lws-operator/lws-controller-manager-7594644fcd-x58df"
Sep 17 18:36:12.492609 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:12.492588    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f72915b1-27d1-41fe-94f6-eac990c3e860-cert\") pod \"lws-controller-manager-7594644fcd-x58df\" (UID: \"f72915b1-27d1-41fe-94f6-eac990c3e860\") " pod="openshift-lws-operator/lws-controller-manager-7594644fcd-x58df"
Sep 17 18:36:12.492702 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:12.492645    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f72915b1-27d1-41fe-94f6-eac990c3e860-metrics-cert\") pod \"lws-controller-manager-7594644fcd-x58df\" (UID: \"f72915b1-27d1-41fe-94f6-eac990c3e860\") " pod="openshift-lws-operator/lws-controller-manager-7594644fcd-x58df"
Sep 17 18:36:12.502765 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:12.502714    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfp6x\" (UniqueName: \"kubernetes.io/projected/f72915b1-27d1-41fe-94f6-eac990c3e860-kube-api-access-gfp6x\") pod \"lws-controller-manager-7594644fcd-x58df\" (UID: \"f72915b1-27d1-41fe-94f6-eac990c3e860\") " pod="openshift-lws-operator/lws-controller-manager-7594644fcd-x58df"
Sep 17 18:36:12.637548 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:12.637497    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7594644fcd-x58df"
Sep 17 18:36:12.914903 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:12.914867    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7594644fcd-x58df"]
Sep 17 18:36:12.918024 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:36:12.917997    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf72915b1_27d1_41fe_94f6_eac990c3e860.slice/crio-4df79ec9ab8d44db0f49c54c6c0a1cacb10de1ec3727f7e7dc82695f6e1f2b09 WatchSource:0}: Error finding container 4df79ec9ab8d44db0f49c54c6c0a1cacb10de1ec3727f7e7dc82695f6e1f2b09: Status 404 returned error can't find the container with id 4df79ec9ab8d44db0f49c54c6c0a1cacb10de1ec3727f7e7dc82695f6e1f2b09
Sep 17 18:36:13.120198 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:13.120165    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7594644fcd-x58df" event={"ID":"f72915b1-27d1-41fe-94f6-eac990c3e860","Type":"ContainerStarted","Data":"4df79ec9ab8d44db0f49c54c6c0a1cacb10de1ec3727f7e7dc82695f6e1f2b09"}
Sep 17 18:36:15.131454 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:15.131412    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7594644fcd-x58df" event={"ID":"f72915b1-27d1-41fe-94f6-eac990c3e860","Type":"ContainerStarted","Data":"4f5e782f32014d8ec8bd9b324ca40b623389023bfeb0e1d911af370c08fffe4b"}
Sep 17 18:36:15.131873 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:15.131554    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7594644fcd-x58df"
Sep 17 18:36:15.151836 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:15.151789    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7594644fcd-x58df" podStartSLOduration=1.433281512 podStartE2EDuration="3.151771225s" totalImagesPullingTime="1.718489713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:36:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:36:15.150543283 +0000 UTC m=+412.001548734" watchObservedRunningTime="2026-09-17 18:36:15.151771225 +0000 UTC m=+412.002776623"
Sep 17 18:36:15.788429 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:15.788382    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:15.894840 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:15.894806    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:17.729775 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:17.729730    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:17.830430 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:17.830401    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:23.818408 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:23.818374    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:26.138311 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:26.138276    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7594644fcd-x58df"
Sep 17 18:36:26.670577 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:26.670540    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:28.343806 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:28.343772    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:28.451069 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:28.451039    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:29.004434 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:29.004403    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:29.109333 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:29.109306    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:29.461345 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:29.461299    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:29.988414 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:29.988357    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:31.354614 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:31.354582    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:31.457223 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:31.457178    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:33.666045 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:33.665985    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:34.009061 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:34.008971    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:36.467459 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:36.467389    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:38.820237 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:38.820206    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:38.919773 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:38.919740    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:40.474831 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:40.474798    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:40.701916 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:40.701886    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:42.276188 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:42.276158    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:42.370597 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:42.370556    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:46.811479 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:46.811428    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:46.913019 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:46.912975    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:47.240057 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:47.239947    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:47.327805 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:47.327757    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:47.604629 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:47.604583    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:47.664086 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:47.664039    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:55.453883 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:55.453834    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:55.545662 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:55.545633    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:58.040933 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:58.040894    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:58.143821 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:36:58.143781    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:01.250062 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:01.250024    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:01.351058 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:01.351023    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:05.659377 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:05.659344    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:05.725394 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:05.725360    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:05.836744 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:05.836699    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:05.937012 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:05.936916    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:06.847432 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:06.847388    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:07.009364 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:07.009315    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:14.440345 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:14.440309    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:14.540174 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:14.540138    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:20.949974 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:20.949943    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:21.050081 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:21.050054    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:21.148688 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:21.148644    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:21.738728 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:21.738697    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-cg76l"]
Sep 17 18:37:21.741802 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:21.741787    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-cg76l"
Sep 17 18:37:21.744014 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:21.743992    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5xpk7\""
Sep 17 18:37:21.744117 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:21.744018    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 18:37:21.744780 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:21.744756    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 18:37:21.749801 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:21.749782    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-cg76l"]
Sep 17 18:37:21.775087 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:21.775058    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:21.826480 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:21.826453    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99p4j\" (UniqueName: \"kubernetes.io/projected/e257d633-c6a3-47b4-ab18-4c7530498188-kube-api-access-99p4j\") pod \"authorino-7bcfd8f9cd-cg76l\" (UID: \"e257d633-c6a3-47b4-ab18-4c7530498188\") " pod="kuadrant-system/authorino-7bcfd8f9cd-cg76l"
Sep 17 18:37:21.862798 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:21.862774    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:21.863478 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:21.863461    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-9267g"]
Sep 17 18:37:21.866585 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:21.866571    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-9267g"
Sep 17 18:37:21.876296 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:21.876279    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-9267g"]
Sep 17 18:37:21.926897 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:21.926873    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99p4j\" (UniqueName: \"kubernetes.io/projected/e257d633-c6a3-47b4-ab18-4c7530498188-kube-api-access-99p4j\") pod \"authorino-7bcfd8f9cd-cg76l\" (UID: \"e257d633-c6a3-47b4-ab18-4c7530498188\") " pod="kuadrant-system/authorino-7bcfd8f9cd-cg76l"
Sep 17 18:37:21.927025 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:21.926933    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sblrh\" (UniqueName: \"kubernetes.io/projected/194efeb4-e038-47b2-8783-a27701d0788c-kube-api-access-sblrh\") pod \"authorino-59c974bd48-9267g\" (UID: \"194efeb4-e038-47b2-8783-a27701d0788c\") " pod="kuadrant-system/authorino-59c974bd48-9267g"
Sep 17 18:37:21.934992 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:21.934966    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99p4j\" (UniqueName: \"kubernetes.io/projected/e257d633-c6a3-47b4-ab18-4c7530498188-kube-api-access-99p4j\") pod \"authorino-7bcfd8f9cd-cg76l\" (UID: \"e257d633-c6a3-47b4-ab18-4c7530498188\") " pod="kuadrant-system/authorino-7bcfd8f9cd-cg76l"
Sep 17 18:37:22.028218 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:22.028117    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sblrh\" (UniqueName: \"kubernetes.io/projected/194efeb4-e038-47b2-8783-a27701d0788c-kube-api-access-sblrh\") pod \"authorino-59c974bd48-9267g\" (UID: \"194efeb4-e038-47b2-8783-a27701d0788c\") " pod="kuadrant-system/authorino-59c974bd48-9267g"
Sep 17 18:37:22.036413 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:22.036392    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sblrh\" (UniqueName: \"kubernetes.io/projected/194efeb4-e038-47b2-8783-a27701d0788c-kube-api-access-sblrh\") pod \"authorino-59c974bd48-9267g\" (UID: \"194efeb4-e038-47b2-8783-a27701d0788c\") " pod="kuadrant-system/authorino-59c974bd48-9267g"
Sep 17 18:37:22.051218 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:22.051197    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-cg76l"
Sep 17 18:37:22.175073 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:22.175031    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-9267g"
Sep 17 18:37:22.323000 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:22.322974    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-cg76l"]
Sep 17 18:37:22.325559 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:37:22.325520    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode257d633_c6a3_47b4_ab18_4c7530498188.slice/crio-24da35c579a9a982565586ae2cd3694e4ab25d2ce1b29c2a4232fd72e36b5d51 WatchSource:0}: Error finding container 24da35c579a9a982565586ae2cd3694e4ab25d2ce1b29c2a4232fd72e36b5d51: Status 404 returned error can't find the container with id 24da35c579a9a982565586ae2cd3694e4ab25d2ce1b29c2a4232fd72e36b5d51
Sep 17 18:37:22.386520 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:22.386478    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-cg76l" event={"ID":"e257d633-c6a3-47b4-ab18-4c7530498188","Type":"ContainerStarted","Data":"24da35c579a9a982565586ae2cd3694e4ab25d2ce1b29c2a4232fd72e36b5d51"}
Sep 17 18:37:22.443136 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:22.443110    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-9267g"]
Sep 17 18:37:22.445479 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:37:22.445453    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod194efeb4_e038_47b2_8783_a27701d0788c.slice/crio-da92cb192ec9c5a68947aca2883909303001906f2102d310b394c91b125b2621 WatchSource:0}: Error finding container da92cb192ec9c5a68947aca2883909303001906f2102d310b394c91b125b2621: Status 404 returned error can't find the container with id da92cb192ec9c5a68947aca2883909303001906f2102d310b394c91b125b2621
Sep 17 18:37:23.393748 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:23.393676    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-9267g" event={"ID":"194efeb4-e038-47b2-8783-a27701d0788c","Type":"ContainerStarted","Data":"da92cb192ec9c5a68947aca2883909303001906f2102d310b394c91b125b2621"}
Sep 17 18:37:24.184033 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:24.183969    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:26.409103 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:26.409068    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-9267g" event={"ID":"194efeb4-e038-47b2-8783-a27701d0788c","Type":"ContainerStarted","Data":"ead2c637cf80fd0201b0d498a42b021693f4b48950037348e6e343d02377f0d4"}
Sep 17 18:37:26.410641 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:26.410615    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-cg76l" event={"ID":"e257d633-c6a3-47b4-ab18-4c7530498188","Type":"ContainerStarted","Data":"494ee8589bf14e150fda443a3c57ce85ee79b52d7a42f0ed92da152d6d79e7d6"}
Sep 17 18:37:26.425377 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:26.425336    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-9267g" podStartSLOduration=2.471906498 podStartE2EDuration="5.425321614s" totalImagesPullingTime="2.953415116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:37:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:37:26.424310335 +0000 UTC m=+483.275315734" watchObservedRunningTime="2026-09-17 18:37:26.425321614 +0000 UTC m=+483.276327012"
Sep 17 18:37:26.439907 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:26.439864    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-cg76l" podStartSLOduration=2.356221714 podStartE2EDuration="5.439850843s" totalImagesPullingTime="3.083629129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:37:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:37:26.43809956 +0000 UTC m=+483.289104960" watchObservedRunningTime="2026-09-17 18:37:26.439850843 +0000 UTC m=+483.290856242"
Sep 17 18:37:26.463776 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:26.463747    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-cg76l"]
Sep 17 18:37:28.418856 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:28.418789    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-cg76l" podUID="e257d633-c6a3-47b4-ab18-4c7530498188" containerName="authorino" containerID="cri-o://494ee8589bf14e150fda443a3c57ce85ee79b52d7a42f0ed92da152d6d79e7d6" gracePeriod=30
Sep 17 18:37:28.654093 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:28.654069    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-cg76l"
Sep 17 18:37:28.787338 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:28.787258    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99p4j\" (UniqueName: \"kubernetes.io/projected/e257d633-c6a3-47b4-ab18-4c7530498188-kube-api-access-99p4j\") pod \"e257d633-c6a3-47b4-ab18-4c7530498188\" (UID: \"e257d633-c6a3-47b4-ab18-4c7530498188\") "
Sep 17 18:37:28.789437 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:28.789411    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e257d633-c6a3-47b4-ab18-4c7530498188-kube-api-access-99p4j" (OuterVolumeSpecName: "kube-api-access-99p4j") pod "e257d633-c6a3-47b4-ab18-4c7530498188" (UID: "e257d633-c6a3-47b4-ab18-4c7530498188"). InnerVolumeSpecName "kube-api-access-99p4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:37:28.888515 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:28.888479    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99p4j\" (UniqueName: \"kubernetes.io/projected/e257d633-c6a3-47b4-ab18-4c7530498188-kube-api-access-99p4j\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:37:29.424268 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:29.424215    2588 generic.go:356] "Generic (PLEG): container finished" podID="e257d633-c6a3-47b4-ab18-4c7530498188" containerID="494ee8589bf14e150fda443a3c57ce85ee79b52d7a42f0ed92da152d6d79e7d6" exitCode=0
Sep 17 18:37:29.424268 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:29.424262    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-cg76l"
Sep 17 18:37:29.424733 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:29.424301    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-cg76l" event={"ID":"e257d633-c6a3-47b4-ab18-4c7530498188","Type":"ContainerDied","Data":"494ee8589bf14e150fda443a3c57ce85ee79b52d7a42f0ed92da152d6d79e7d6"}
Sep 17 18:37:29.424733 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:29.424343    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-cg76l" event={"ID":"e257d633-c6a3-47b4-ab18-4c7530498188","Type":"ContainerDied","Data":"24da35c579a9a982565586ae2cd3694e4ab25d2ce1b29c2a4232fd72e36b5d51"}
Sep 17 18:37:29.424733 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:29.424360    2588 scope.go:117] "RemoveContainer" containerID="494ee8589bf14e150fda443a3c57ce85ee79b52d7a42f0ed92da152d6d79e7d6"
Sep 17 18:37:29.433104 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:29.433088    2588 scope.go:117] "RemoveContainer" containerID="494ee8589bf14e150fda443a3c57ce85ee79b52d7a42f0ed92da152d6d79e7d6"
Sep 17 18:37:29.433356 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:37:29.433341    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"494ee8589bf14e150fda443a3c57ce85ee79b52d7a42f0ed92da152d6d79e7d6\": container with ID starting with 494ee8589bf14e150fda443a3c57ce85ee79b52d7a42f0ed92da152d6d79e7d6 not found: ID does not exist" containerID="494ee8589bf14e150fda443a3c57ce85ee79b52d7a42f0ed92da152d6d79e7d6"
Sep 17 18:37:29.433413 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:29.433363    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"494ee8589bf14e150fda443a3c57ce85ee79b52d7a42f0ed92da152d6d79e7d6"} err="failed to get container status \"494ee8589bf14e150fda443a3c57ce85ee79b52d7a42f0ed92da152d6d79e7d6\": rpc error: code = NotFound desc = could not find container \"494ee8589bf14e150fda443a3c57ce85ee79b52d7a42f0ed92da152d6d79e7d6\": container with ID starting with 494ee8589bf14e150fda443a3c57ce85ee79b52d7a42f0ed92da152d6d79e7d6 not found: ID does not exist"
Sep 17 18:37:29.446490 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:29.446469    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-cg76l"]
Sep 17 18:37:29.450887 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:29.450862    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-cg76l"]
Sep 17 18:37:29.505950 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:29.505925    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e257d633-c6a3-47b4-ab18-4c7530498188" path="/var/lib/kubelet/pods/e257d633-c6a3-47b4-ab18-4c7530498188/volumes"
Sep 17 18:37:41.072583 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:41.072549    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:41.396958 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:41.396869    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:44.027246 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:44.027208    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d9898bd99-6hlmm"]
Sep 17 18:37:44.027647 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:44.027501    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e257d633-c6a3-47b4-ab18-4c7530498188" containerName="authorino"
Sep 17 18:37:44.027647 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:44.027513    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="e257d633-c6a3-47b4-ab18-4c7530498188" containerName="authorino"
Sep 17 18:37:44.027647 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:44.027595    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="e257d633-c6a3-47b4-ab18-4c7530498188" containerName="authorino"
Sep 17 18:37:44.031681 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:44.031654    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-6hlmm"
Sep 17 18:37:44.034301 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:44.034280    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 17 18:37:44.041586 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:44.041558    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-6hlmm"]
Sep 17 18:37:44.217381 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:44.217337    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/067e3a44-769a-4ad5-87fb-e25fd9430437-tls-cert\") pod \"authorino-6d9898bd99-6hlmm\" (UID: \"067e3a44-769a-4ad5-87fb-e25fd9430437\") " pod="kuadrant-system/authorino-6d9898bd99-6hlmm"
Sep 17 18:37:44.217649 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:44.217403    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmmzw\" (UniqueName: \"kubernetes.io/projected/067e3a44-769a-4ad5-87fb-e25fd9430437-kube-api-access-pmmzw\") pod \"authorino-6d9898bd99-6hlmm\" (UID: \"067e3a44-769a-4ad5-87fb-e25fd9430437\") " pod="kuadrant-system/authorino-6d9898bd99-6hlmm"
Sep 17 18:37:44.318547 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:44.318480    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/067e3a44-769a-4ad5-87fb-e25fd9430437-tls-cert\") pod \"authorino-6d9898bd99-6hlmm\" (UID: \"067e3a44-769a-4ad5-87fb-e25fd9430437\") " pod="kuadrant-system/authorino-6d9898bd99-6hlmm"
Sep 17 18:37:44.318711 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:44.318582    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmmzw\" (UniqueName: \"kubernetes.io/projected/067e3a44-769a-4ad5-87fb-e25fd9430437-kube-api-access-pmmzw\") pod \"authorino-6d9898bd99-6hlmm\" (UID: \"067e3a44-769a-4ad5-87fb-e25fd9430437\") " pod="kuadrant-system/authorino-6d9898bd99-6hlmm"
Sep 17 18:37:44.320914 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:44.320892    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/067e3a44-769a-4ad5-87fb-e25fd9430437-tls-cert\") pod \"authorino-6d9898bd99-6hlmm\" (UID: \"067e3a44-769a-4ad5-87fb-e25fd9430437\") " pod="kuadrant-system/authorino-6d9898bd99-6hlmm"
Sep 17 18:37:44.328078 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:44.328056    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmmzw\" (UniqueName: \"kubernetes.io/projected/067e3a44-769a-4ad5-87fb-e25fd9430437-kube-api-access-pmmzw\") pod \"authorino-6d9898bd99-6hlmm\" (UID: \"067e3a44-769a-4ad5-87fb-e25fd9430437\") " pod="kuadrant-system/authorino-6d9898bd99-6hlmm"
Sep 17 18:37:44.341978 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:44.341960    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-6hlmm"
Sep 17 18:37:44.617296 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:44.617224    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-6hlmm"]
Sep 17 18:37:44.620160 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:37:44.620122    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod067e3a44_769a_4ad5_87fb_e25fd9430437.slice/crio-a511e5c35e8aa70b343a64c58ef1d4b03710a668811202f279c96bc6b906bfb1 WatchSource:0}: Error finding container a511e5c35e8aa70b343a64c58ef1d4b03710a668811202f279c96bc6b906bfb1: Status 404 returned error can't find the container with id a511e5c35e8aa70b343a64c58ef1d4b03710a668811202f279c96bc6b906bfb1
Sep 17 18:37:45.489373 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:45.489289    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-6hlmm" event={"ID":"067e3a44-769a-4ad5-87fb-e25fd9430437","Type":"ContainerStarted","Data":"529bd4e0caeca3a268a1c641d5577a2144520171ea401e7bcacfdb27c314d8b3"}
Sep 17 18:37:45.489373 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:45.489330    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-6hlmm" event={"ID":"067e3a44-769a-4ad5-87fb-e25fd9430437","Type":"ContainerStarted","Data":"a511e5c35e8aa70b343a64c58ef1d4b03710a668811202f279c96bc6b906bfb1"}
Sep 17 18:37:45.505078 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:45.505034    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d9898bd99-6hlmm" podStartSLOduration=0.986841645 podStartE2EDuration="1.505018645s" totalImagesPullingTime="518.177ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:37:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:37:45.504063731 +0000 UTC m=+502.355069132" watchObservedRunningTime="2026-09-17 18:37:45.505018645 +0000 UTC m=+502.356024043"
Sep 17 18:37:45.533789 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:45.533758    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-9267g"]
Sep 17 18:37:45.533949 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:45.533922    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-9267g" podUID="194efeb4-e038-47b2-8783-a27701d0788c" containerName="authorino" containerID="cri-o://ead2c637cf80fd0201b0d498a42b021693f4b48950037348e6e343d02377f0d4" gracePeriod=30
Sep 17 18:37:45.781675 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:45.781649    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-9267g"
Sep 17 18:37:45.933692 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:45.933663    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sblrh\" (UniqueName: \"kubernetes.io/projected/194efeb4-e038-47b2-8783-a27701d0788c-kube-api-access-sblrh\") pod \"194efeb4-e038-47b2-8783-a27701d0788c\" (UID: \"194efeb4-e038-47b2-8783-a27701d0788c\") "
Sep 17 18:37:45.935744 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:45.935715    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/194efeb4-e038-47b2-8783-a27701d0788c-kube-api-access-sblrh" (OuterVolumeSpecName: "kube-api-access-sblrh") pod "194efeb4-e038-47b2-8783-a27701d0788c" (UID: "194efeb4-e038-47b2-8783-a27701d0788c"). InnerVolumeSpecName "kube-api-access-sblrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:37:46.034236 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:46.034151    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sblrh\" (UniqueName: \"kubernetes.io/projected/194efeb4-e038-47b2-8783-a27701d0788c-kube-api-access-sblrh\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:37:46.494463 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:46.494424    2588 generic.go:356] "Generic (PLEG): container finished" podID="194efeb4-e038-47b2-8783-a27701d0788c" containerID="ead2c637cf80fd0201b0d498a42b021693f4b48950037348e6e343d02377f0d4" exitCode=0
Sep 17 18:37:46.494892 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:46.494469    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-9267g"
Sep 17 18:37:46.494892 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:46.494507    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-9267g" event={"ID":"194efeb4-e038-47b2-8783-a27701d0788c","Type":"ContainerDied","Data":"ead2c637cf80fd0201b0d498a42b021693f4b48950037348e6e343d02377f0d4"}
Sep 17 18:37:46.494892 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:46.494564    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-9267g" event={"ID":"194efeb4-e038-47b2-8783-a27701d0788c","Type":"ContainerDied","Data":"da92cb192ec9c5a68947aca2883909303001906f2102d310b394c91b125b2621"}
Sep 17 18:37:46.494892 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:46.494595    2588 scope.go:117] "RemoveContainer" containerID="ead2c637cf80fd0201b0d498a42b021693f4b48950037348e6e343d02377f0d4"
Sep 17 18:37:46.502567 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:46.502550    2588 scope.go:117] "RemoveContainer" containerID="ead2c637cf80fd0201b0d498a42b021693f4b48950037348e6e343d02377f0d4"
Sep 17 18:37:46.502834 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:37:46.502812    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ead2c637cf80fd0201b0d498a42b021693f4b48950037348e6e343d02377f0d4\": container with ID starting with ead2c637cf80fd0201b0d498a42b021693f4b48950037348e6e343d02377f0d4 not found: ID does not exist" containerID="ead2c637cf80fd0201b0d498a42b021693f4b48950037348e6e343d02377f0d4"
Sep 17 18:37:46.502923 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:46.502842    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ead2c637cf80fd0201b0d498a42b021693f4b48950037348e6e343d02377f0d4"} err="failed to get container status \"ead2c637cf80fd0201b0d498a42b021693f4b48950037348e6e343d02377f0d4\": rpc error: code = NotFound desc = could not find container \"ead2c637cf80fd0201b0d498a42b021693f4b48950037348e6e343d02377f0d4\": container with ID starting with ead2c637cf80fd0201b0d498a42b021693f4b48950037348e6e343d02377f0d4 not found: ID does not exist"
Sep 17 18:37:46.515470 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:46.515448    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-9267g"]
Sep 17 18:37:46.516657 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:46.516639    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-9267g"]
Sep 17 18:37:47.504941 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:47.504910    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="194efeb4-e038-47b2-8783-a27701d0788c" path="/var/lib/kubelet/pods/194efeb4-e038-47b2-8783-a27701d0788c/volumes"
Sep 17 18:37:49.408463 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:49.408423    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:49.508389 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:37:49.508358    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:16.447382 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:38:16.447291    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:16.519998 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:38:16.519959    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:16.638838 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:38:16.638807    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:16.739657 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:38:16.739580    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:16.841944 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:38:16.841896    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:17.122361 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:38:17.122330    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:17.199995 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:38:17.199962    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:05.960393 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:39:05.960363    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:06.023015 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:39:06.022988    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:09.668393 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:39:09.668365    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:09.765607 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:39:09.765573    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:25.465851 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:39:25.465820    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xlklb_07ee31e0-43bb-4940-837d-969bd840492b/console-operator/1.log"
Sep 17 18:39:25.470192 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:39:25.470173    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xlklb_07ee31e0-43bb-4940-837d-969bd840492b/console-operator/1.log"
Sep 17 18:39:25.471077 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:39:25.471056    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovn-acl-logging/0.log"
Sep 17 18:39:25.475584 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:39:25.475564    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovn-acl-logging/0.log"
Sep 17 18:39:26.862429 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:39:26.862399    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:30.791102 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:39:30.791052    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:30.890345 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:39:30.890309    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:34.752096 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:39:34.752059    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:34.852688 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:39:34.852655    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:35.020722 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:39:35.020618    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:35.120123 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:39:35.120082    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:35.219678 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:39:35.219645    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:35.946915 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:39:35.946884    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:36.048447 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:39:36.048357    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:38.599623 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:39:38.599590    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:00.796200 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:00.796160    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:10.601589 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:10.601478    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:10.696922 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:10.696890    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:23.352813 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:23.352775    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:23.550570 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:23.550509    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:31.439136 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:31.439094    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:31.824543 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:31.824495    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb"]
Sep 17 18:41:31.824820 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:31.824807    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="194efeb4-e038-47b2-8783-a27701d0788c" containerName="authorino"
Sep 17 18:41:31.824864 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:31.824822    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="194efeb4-e038-47b2-8783-a27701d0788c" containerName="authorino"
Sep 17 18:41:31.824898 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:31.824869    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="194efeb4-e038-47b2-8783-a27701d0788c" containerName="authorino"
Sep 17 18:41:31.827721 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:31.827705    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb"
Sep 17 18:41:31.832710 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:31.832674    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-k4ft2\""
Sep 17 18:41:31.833479 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:31.833453    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 17 18:41:31.834109 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:31.833756    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 17 18:41:31.834109 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:31.833799    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 17 18:41:31.836718 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:31.836701    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-dgs77\""
Sep 17 18:41:31.856941 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:31.856917    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb"]
Sep 17 18:41:31.873092 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:31.873071    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:31.971886 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:31.971842    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/105896b8-cb7b-4279-a33d-4c25c8f72f52-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb\" (UID: \"105896b8-cb7b-4279-a33d-4c25c8f72f52\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb"
Sep 17 18:41:31.972058 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:31.971962    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfcqk\" (UniqueName: \"kubernetes.io/projected/105896b8-cb7b-4279-a33d-4c25c8f72f52-kube-api-access-kfcqk\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb\" (UID: \"105896b8-cb7b-4279-a33d-4c25c8f72f52\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb"
Sep 17 18:41:32.073233 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:32.073194    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfcqk\" (UniqueName: \"kubernetes.io/projected/105896b8-cb7b-4279-a33d-4c25c8f72f52-kube-api-access-kfcqk\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb\" (UID: \"105896b8-cb7b-4279-a33d-4c25c8f72f52\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb"
Sep 17 18:41:32.073410 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:32.073246    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/105896b8-cb7b-4279-a33d-4c25c8f72f52-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb\" (UID: \"105896b8-cb7b-4279-a33d-4c25c8f72f52\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb"
Sep 17 18:41:32.075828 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:32.075765    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/105896b8-cb7b-4279-a33d-4c25c8f72f52-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb\" (UID: \"105896b8-cb7b-4279-a33d-4c25c8f72f52\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb"
Sep 17 18:41:32.082295 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:32.082275    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfcqk\" (UniqueName: \"kubernetes.io/projected/105896b8-cb7b-4279-a33d-4c25c8f72f52-kube-api-access-kfcqk\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb\" (UID: \"105896b8-cb7b-4279-a33d-4c25c8f72f52\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb"
Sep 17 18:41:32.137263 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:32.137232    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb"
Sep 17 18:41:32.413023 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:32.412940    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb"]
Sep 17 18:41:32.416193 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:41:32.416168    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod105896b8_cb7b_4279_a33d_4c25c8f72f52.slice/crio-dd0b2b41740b5243688123c0ee2da0555196e91f43795a8735f86d8295947e08 WatchSource:0}: Error finding container dd0b2b41740b5243688123c0ee2da0555196e91f43795a8735f86d8295947e08: Status 404 returned error can't find the container with id dd0b2b41740b5243688123c0ee2da0555196e91f43795a8735f86d8295947e08
Sep 17 18:41:32.418899 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:32.418882    2588 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:41:33.384140 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:33.384096    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb" event={"ID":"105896b8-cb7b-4279-a33d-4c25c8f72f52","Type":"ContainerStarted","Data":"dd0b2b41740b5243688123c0ee2da0555196e91f43795a8735f86d8295947e08"}
Sep 17 18:41:36.398960 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:36.398916    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb" event={"ID":"105896b8-cb7b-4279-a33d-4c25c8f72f52","Type":"ContainerStarted","Data":"1aa1fa691871690a3b0816e7c99e947f018360bb0774db6d369f3df57aae9116"}
Sep 17 18:41:36.399348 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:36.399059    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb"
Sep 17 18:41:41.422660 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:41.422578    2588 generic.go:356] "Generic (PLEG): container finished" podID="585c7370-1cb3-4035-ac56-be5c3f566027" containerID="49b7be038103af0e31313793fbfa9a34dbb46034e2c0a684030d8e1cb02f078e" exitCode=0
Sep 17 18:41:41.422660 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:41.422645    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k" event={"ID":"585c7370-1cb3-4035-ac56-be5c3f566027","Type":"ContainerDied","Data":"49b7be038103af0e31313793fbfa9a34dbb46034e2c0a684030d8e1cb02f078e"}
Sep 17 18:41:41.423105 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:41.423086    2588 scope.go:117] "RemoveContainer" containerID="49b7be038103af0e31313793fbfa9a34dbb46034e2c0a684030d8e1cb02f078e"
Sep 17 18:41:41.424160 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:41.424133    2588 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 18:41:41.424214 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:41.424200    2588 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 18:41:41.424257 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:41.424238    2588 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 18:41:41.447239 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:41.447193    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb" podStartSLOduration=7.514587441 podStartE2EDuration="10.447176289s" totalImagesPullingTime="2.932588848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:41:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:41:36.424490685 +0000 UTC m=+733.275496085" watchObservedRunningTime="2026-09-17 18:41:41.447176289 +0000 UTC m=+738.298181690"
Sep 17 18:41:42.429944 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:42.429901    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k" event={"ID":"585c7370-1cb3-4035-ac56-be5c3f566027","Type":"ContainerStarted","Data":"388a1af5245a937e0a54520747b3c66ecbf0b1f6be3095513e9ae17d76d5e873"}
Sep 17 18:41:43.257592 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:43.257557    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:41:43.257774 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:43.257747    2588 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-72h4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body=
Sep 17 18:41:43.257826 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:43.257797    2588 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k" podUID="585c7370-1cb3-4035-ac56-be5c3f566027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused"
Sep 17 18:41:44.257713 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:44.257678    2588 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-72h4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body=
Sep 17 18:41:44.258205 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:44.257749    2588 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k" podUID="585c7370-1cb3-4035-ac56-be5c3f566027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused"
Sep 17 18:41:45.258012 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:45.257977    2588 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-72h4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body=
Sep 17 18:41:45.258400 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:45.258033    2588 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k" podUID="585c7370-1cb3-4035-ac56-be5c3f566027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused"
Sep 17 18:41:46.257736 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:46.257693    2588 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-72h4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body=
Sep 17 18:41:46.257916 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:46.257760    2588 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k" podUID="585c7370-1cb3-4035-ac56-be5c3f566027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused"
Sep 17 18:41:47.257653 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:47.257619    2588 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-72h4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body=
Sep 17 18:41:47.258047 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:47.257677    2588 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k" podUID="585c7370-1cb3-4035-ac56-be5c3f566027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused"
Sep 17 18:41:48.260844 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:48.260815    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:41:48.261230 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:48.261111    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:41:48.261679 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:48.261664    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-72h4k"
Sep 17 18:41:53.853957 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:53.853923    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:54.069167 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:54.069105    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g"]
Sep 17 18:41:54.074054 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:54.074027    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g"
Sep 17 18:41:54.077303 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:54.077277    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 17 18:41:54.077647 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:54.077604    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 17 18:41:54.077647 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:54.077624    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-6t2zd\""
Sep 17 18:41:54.077989 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:54.077970    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-pcrwh\""
Sep 17 18:41:54.077989 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:54.078184    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 17 18:41:54.083288 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:54.083266    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g"]
Sep 17 18:41:54.112366 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:54.112298    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:54.178449 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:54.178385    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75720580-4c37-44fa-8bc3-83c4549ed1c6-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g\" (UID: \"75720580-4c37-44fa-8bc3-83c4549ed1c6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g"
Sep 17 18:41:54.178668 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:54.178483    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdsmb\" (UniqueName: \"kubernetes.io/projected/75720580-4c37-44fa-8bc3-83c4549ed1c6-kube-api-access-vdsmb\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g\" (UID: \"75720580-4c37-44fa-8bc3-83c4549ed1c6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g"
Sep 17 18:41:54.278937 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:54.278903    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75720580-4c37-44fa-8bc3-83c4549ed1c6-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g\" (UID: \"75720580-4c37-44fa-8bc3-83c4549ed1c6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g"
Sep 17 18:41:54.279099 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:54.278956    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdsmb\" (UniqueName: \"kubernetes.io/projected/75720580-4c37-44fa-8bc3-83c4549ed1c6-kube-api-access-vdsmb\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g\" (UID: \"75720580-4c37-44fa-8bc3-83c4549ed1c6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g"
Sep 17 18:41:54.281458 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:54.281436    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75720580-4c37-44fa-8bc3-83c4549ed1c6-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g\" (UID: \"75720580-4c37-44fa-8bc3-83c4549ed1c6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g"
Sep 17 18:41:54.288575 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:54.288553    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdsmb\" (UniqueName: \"kubernetes.io/projected/75720580-4c37-44fa-8bc3-83c4549ed1c6-kube-api-access-vdsmb\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g\" (UID: \"75720580-4c37-44fa-8bc3-83c4549ed1c6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g"
Sep 17 18:41:54.385941 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:54.385861    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g"
Sep 17 18:41:54.667970 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:54.666452    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g"]
Sep 17 18:41:54.669517 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:41:54.669482    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75720580_4c37_44fa_8bc3_83c4549ed1c6.slice/crio-d0a4eb0d0b6146cb5417f2e86895762289c4e85deb473157ef67fb70f5a05d27 WatchSource:0}: Error finding container d0a4eb0d0b6146cb5417f2e86895762289c4e85deb473157ef67fb70f5a05d27: Status 404 returned error can't find the container with id d0a4eb0d0b6146cb5417f2e86895762289c4e85deb473157ef67fb70f5a05d27
Sep 17 18:41:55.482926 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:55.482893    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g" event={"ID":"75720580-4c37-44fa-8bc3-83c4549ed1c6","Type":"ContainerStarted","Data":"5d7b100982658d549b3f597e32543e7bc822cadcabebe0a25708870ab5761d4a"}
Sep 17 18:41:55.483343 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:55.482987    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g" event={"ID":"75720580-4c37-44fa-8bc3-83c4549ed1c6","Type":"ContainerStarted","Data":"d0a4eb0d0b6146cb5417f2e86895762289c4e85deb473157ef67fb70f5a05d27"}
Sep 17 18:41:55.483343 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:55.483016    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g"
Sep 17 18:41:55.502869 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:41:55.502822    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g" podStartSLOduration=1.502805333 podStartE2EDuration="1.502805333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:41:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:41:55.502415981 +0000 UTC m=+752.353421381" watchObservedRunningTime="2026-09-17 18:41:55.502805333 +0000 UTC m=+752.353810731"
Sep 17 18:42:07.406720 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:07.406691    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb"
Sep 17 18:42:09.623376 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:09.623346    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:19.256347 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:19.256307    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:26.491277 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:26.491245    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g"
Sep 17 18:42:28.354957 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:28.354911    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:28.365470 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:28.365432    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb"]
Sep 17 18:42:28.500460 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:28.500428    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:29.132478 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:29.132442    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:29.144693 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:29.144661    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g"]
Sep 17 18:42:29.211795 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:29.211759    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:36.490894 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:36.490812    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g" podUID="75720580-4c37-44fa-8bc3-83c4549ed1c6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:42:37.406779 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:37.406742    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb" podUID="105896b8-cb7b-4279-a33d-4c25c8f72f52" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:42:40.982864 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:40.982821    2588 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g" message="" err="command '/bin/sleep 15' exited with 137: "
Sep 17 18:42:40.983197 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:42:40.982865    2588 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g" podUID="75720580-4c37-44fa-8bc3-83c4549ed1c6" containerName="main" containerID="cri-o://5d7b100982658d549b3f597e32543e7bc822cadcabebe0a25708870ab5761d4a"
Sep 17 18:42:40.983197 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:40.982925    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g" podUID="75720580-4c37-44fa-8bc3-83c4549ed1c6" containerName="main" containerID="cri-o://5d7b100982658d549b3f597e32543e7bc822cadcabebe0a25708870ab5761d4a" gracePeriod=49
Sep 17 18:42:41.125399 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:41.125376    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g_75720580-4c37-44fa-8bc3-83c4549ed1c6/main/0.log"
Sep 17 18:42:41.127711 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:41.127695    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g"
Sep 17 18:42:41.165664 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:41.165633    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdsmb\" (UniqueName: \"kubernetes.io/projected/75720580-4c37-44fa-8bc3-83c4549ed1c6-kube-api-access-vdsmb\") pod \"75720580-4c37-44fa-8bc3-83c4549ed1c6\" (UID: \"75720580-4c37-44fa-8bc3-83c4549ed1c6\") "
Sep 17 18:42:41.165813 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:41.165697    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75720580-4c37-44fa-8bc3-83c4549ed1c6-tls-certs\") pod \"75720580-4c37-44fa-8bc3-83c4549ed1c6\" (UID: \"75720580-4c37-44fa-8bc3-83c4549ed1c6\") "
Sep 17 18:42:41.167879 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:41.167848    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75720580-4c37-44fa-8bc3-83c4549ed1c6-kube-api-access-vdsmb" (OuterVolumeSpecName: "kube-api-access-vdsmb") pod "75720580-4c37-44fa-8bc3-83c4549ed1c6" (UID: "75720580-4c37-44fa-8bc3-83c4549ed1c6"). InnerVolumeSpecName "kube-api-access-vdsmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:42:41.167989 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:41.167857    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75720580-4c37-44fa-8bc3-83c4549ed1c6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "75720580-4c37-44fa-8bc3-83c4549ed1c6" (UID: "75720580-4c37-44fa-8bc3-83c4549ed1c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:42:41.267221 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:41.267144    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdsmb\" (UniqueName: \"kubernetes.io/projected/75720580-4c37-44fa-8bc3-83c4549ed1c6-kube-api-access-vdsmb\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:42:41.267221 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:41.267180    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75720580-4c37-44fa-8bc3-83c4549ed1c6-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:42:41.670335 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:41.670304    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g_75720580-4c37-44fa-8bc3-83c4549ed1c6/main/0.log"
Sep 17 18:42:41.670499 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:41.670354    2588 generic.go:356] "Generic (PLEG): container finished" podID="75720580-4c37-44fa-8bc3-83c4549ed1c6" containerID="5d7b100982658d549b3f597e32543e7bc822cadcabebe0a25708870ab5761d4a" exitCode=1
Sep 17 18:42:41.670499 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:41.670404    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g"
Sep 17 18:42:41.670499 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:41.670427    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g" event={"ID":"75720580-4c37-44fa-8bc3-83c4549ed1c6","Type":"ContainerDied","Data":"5d7b100982658d549b3f597e32543e7bc822cadcabebe0a25708870ab5761d4a"}
Sep 17 18:42:41.670499 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:41.670493    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g" event={"ID":"75720580-4c37-44fa-8bc3-83c4549ed1c6","Type":"ContainerDied","Data":"d0a4eb0d0b6146cb5417f2e86895762289c4e85deb473157ef67fb70f5a05d27"}
Sep 17 18:42:41.670667 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:41.670546    2588 scope.go:117] "RemoveContainer" containerID="5d7b100982658d549b3f597e32543e7bc822cadcabebe0a25708870ab5761d4a"
Sep 17 18:42:41.678427 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:41.678411    2588 scope.go:117] "RemoveContainer" containerID="5d7b100982658d549b3f597e32543e7bc822cadcabebe0a25708870ab5761d4a"
Sep 17 18:42:41.678791 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:42:41.678773    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d7b100982658d549b3f597e32543e7bc822cadcabebe0a25708870ab5761d4a\": container with ID starting with 5d7b100982658d549b3f597e32543e7bc822cadcabebe0a25708870ab5761d4a not found: ID does not exist" containerID="5d7b100982658d549b3f597e32543e7bc822cadcabebe0a25708870ab5761d4a"
Sep 17 18:42:41.678877 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:41.678798    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d7b100982658d549b3f597e32543e7bc822cadcabebe0a25708870ab5761d4a"} err="failed to get container status \"5d7b100982658d549b3f597e32543e7bc822cadcabebe0a25708870ab5761d4a\": rpc error: code = NotFound desc = could not find container \"5d7b100982658d549b3f597e32543e7bc822cadcabebe0a25708870ab5761d4a\": container with ID starting with 5d7b100982658d549b3f597e32543e7bc822cadcabebe0a25708870ab5761d4a not found: ID does not exist"
Sep 17 18:42:41.689071 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:41.689050    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g"]
Sep 17 18:42:41.692389 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:41.692368    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cs72g"]
Sep 17 18:42:43.385763 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:43.385715    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb" podUID="105896b8-cb7b-4279-a33d-4c25c8f72f52" containerName="main" containerID="cri-o://1aa1fa691871690a3b0816e7c99e947f018360bb0774db6d369f3df57aae9116" gracePeriod=45
Sep 17 18:42:43.506905 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:43.506858    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75720580-4c37-44fa-8bc3-83c4549ed1c6" path="/var/lib/kubelet/pods/75720580-4c37-44fa-8bc3-83c4549ed1c6/volumes"
Sep 17 18:42:47.407236 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:47.407199    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb" podUID="105896b8-cb7b-4279-a33d-4c25c8f72f52" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:42:57.406932 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:57.406889    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb" podUID="105896b8-cb7b-4279-a33d-4c25c8f72f52" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:42:57.407390 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:42:57.407034    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb"
Sep 17 18:43:07.406731 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:07.406694    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb" podUID="105896b8-cb7b-4279-a33d-4c25c8f72f52" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:43:13.631396 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:13.631363    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb"
Sep 17 18:43:13.732324 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:13.732246    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/105896b8-cb7b-4279-a33d-4c25c8f72f52-tls-certs\") pod \"105896b8-cb7b-4279-a33d-4c25c8f72f52\" (UID: \"105896b8-cb7b-4279-a33d-4c25c8f72f52\") "
Sep 17 18:43:13.732324 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:13.732283    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfcqk\" (UniqueName: \"kubernetes.io/projected/105896b8-cb7b-4279-a33d-4c25c8f72f52-kube-api-access-kfcqk\") pod \"105896b8-cb7b-4279-a33d-4c25c8f72f52\" (UID: \"105896b8-cb7b-4279-a33d-4c25c8f72f52\") "
Sep 17 18:43:13.734564 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:13.734521    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/105896b8-cb7b-4279-a33d-4c25c8f72f52-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "105896b8-cb7b-4279-a33d-4c25c8f72f52" (UID: "105896b8-cb7b-4279-a33d-4c25c8f72f52"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:43:13.734690 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:13.734578    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/105896b8-cb7b-4279-a33d-4c25c8f72f52-kube-api-access-kfcqk" (OuterVolumeSpecName: "kube-api-access-kfcqk") pod "105896b8-cb7b-4279-a33d-4c25c8f72f52" (UID: "105896b8-cb7b-4279-a33d-4c25c8f72f52"). InnerVolumeSpecName "kube-api-access-kfcqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:43:13.797062 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:13.797027    2588 generic.go:356] "Generic (PLEG): container finished" podID="105896b8-cb7b-4279-a33d-4c25c8f72f52" containerID="1aa1fa691871690a3b0816e7c99e947f018360bb0774db6d369f3df57aae9116" exitCode=0
Sep 17 18:43:13.797195 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:13.797070    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb"
Sep 17 18:43:13.797195 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:13.797090    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb" event={"ID":"105896b8-cb7b-4279-a33d-4c25c8f72f52","Type":"ContainerDied","Data":"1aa1fa691871690a3b0816e7c99e947f018360bb0774db6d369f3df57aae9116"}
Sep 17 18:43:13.797195 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:13.797136    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb" event={"ID":"105896b8-cb7b-4279-a33d-4c25c8f72f52","Type":"ContainerDied","Data":"dd0b2b41740b5243688123c0ee2da0555196e91f43795a8735f86d8295947e08"}
Sep 17 18:43:13.797195 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:13.797161    2588 scope.go:117] "RemoveContainer" containerID="1aa1fa691871690a3b0816e7c99e947f018360bb0774db6d369f3df57aae9116"
Sep 17 18:43:13.805388 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:13.805370    2588 scope.go:117] "RemoveContainer" containerID="1aa1fa691871690a3b0816e7c99e947f018360bb0774db6d369f3df57aae9116"
Sep 17 18:43:13.805692 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:43:13.805672    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aa1fa691871690a3b0816e7c99e947f018360bb0774db6d369f3df57aae9116\": container with ID starting with 1aa1fa691871690a3b0816e7c99e947f018360bb0774db6d369f3df57aae9116 not found: ID does not exist" containerID="1aa1fa691871690a3b0816e7c99e947f018360bb0774db6d369f3df57aae9116"
Sep 17 18:43:13.805757 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:13.805699    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aa1fa691871690a3b0816e7c99e947f018360bb0774db6d369f3df57aae9116"} err="failed to get container status \"1aa1fa691871690a3b0816e7c99e947f018360bb0774db6d369f3df57aae9116\": rpc error: code = NotFound desc = could not find container \"1aa1fa691871690a3b0816e7c99e947f018360bb0774db6d369f3df57aae9116\": container with ID starting with 1aa1fa691871690a3b0816e7c99e947f018360bb0774db6d369f3df57aae9116 not found: ID does not exist"
Sep 17 18:43:13.817635 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:13.817615    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb"]
Sep 17 18:43:13.822183 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:13.822163    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-hmqpb"]
Sep 17 18:43:13.833075 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:13.833056    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/105896b8-cb7b-4279-a33d-4c25c8f72f52-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:43:13.833137 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:13.833078    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfcqk\" (UniqueName: \"kubernetes.io/projected/105896b8-cb7b-4279-a33d-4c25c8f72f52-kube-api-access-kfcqk\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:43:15.505297 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:15.505255    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="105896b8-cb7b-4279-a33d-4c25c8f72f52" path="/var/lib/kubelet/pods/105896b8-cb7b-4279-a33d-4c25c8f72f52/volumes"
Sep 17 18:43:17.241521 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:17.241489    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:17.355605 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:17.355522    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:17.556839 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:17.556804    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:17.653588 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:17.653556    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:22.045469 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:22.045439    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:22.309333 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:22.309304    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh"]
Sep 17 18:43:22.309609 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:22.309596    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75720580-4c37-44fa-8bc3-83c4549ed1c6" containerName="main"
Sep 17 18:43:22.309663 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:22.309611    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="75720580-4c37-44fa-8bc3-83c4549ed1c6" containerName="main"
Sep 17 18:43:22.309663 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:22.309619    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="105896b8-cb7b-4279-a33d-4c25c8f72f52" containerName="main"
Sep 17 18:43:22.309663 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:22.309624    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="105896b8-cb7b-4279-a33d-4c25c8f72f52" containerName="main"
Sep 17 18:43:22.309788 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:22.309682    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="105896b8-cb7b-4279-a33d-4c25c8f72f52" containerName="main"
Sep 17 18:43:22.309788 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:22.309690    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="75720580-4c37-44fa-8bc3-83c4549ed1c6" containerName="main"
Sep 17 18:43:22.314245 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:22.314221    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh"
Sep 17 18:43:22.317332 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:22.317303    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-qh8vv\""
Sep 17 18:43:22.317522 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:22.317498    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 17 18:43:22.317678 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:22.317347    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-sn2kx\""
Sep 17 18:43:22.317678 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:22.317551    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 17 18:43:22.317678 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:22.317347    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 17 18:43:22.330944 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:22.330923    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh"]
Sep 17 18:43:22.356442 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:22.356418    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:22.405188 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:22.405161    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75c8r\" (UniqueName: \"kubernetes.io/projected/d20b1e1e-b37d-451c-8540-0465e0a1aa9f-kube-api-access-75c8r\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh\" (UID: \"d20b1e1e-b37d-451c-8540-0465e0a1aa9f\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh"
Sep 17 18:43:22.405311 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:22.405221    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20b1e1e-b37d-451c-8540-0465e0a1aa9f-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh\" (UID: \"d20b1e1e-b37d-451c-8540-0465e0a1aa9f\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh"
Sep 17 18:43:22.506105 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:22.506073    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20b1e1e-b37d-451c-8540-0465e0a1aa9f-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh\" (UID: \"d20b1e1e-b37d-451c-8540-0465e0a1aa9f\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh"
Sep 17 18:43:22.506293 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:22.506127    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75c8r\" (UniqueName: \"kubernetes.io/projected/d20b1e1e-b37d-451c-8540-0465e0a1aa9f-kube-api-access-75c8r\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh\" (UID: \"d20b1e1e-b37d-451c-8540-0465e0a1aa9f\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh"
Sep 17 18:43:22.508631 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:22.508608    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20b1e1e-b37d-451c-8540-0465e0a1aa9f-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh\" (UID: \"d20b1e1e-b37d-451c-8540-0465e0a1aa9f\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh"
Sep 17 18:43:22.513942 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:22.513919    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75c8r\" (UniqueName: \"kubernetes.io/projected/d20b1e1e-b37d-451c-8540-0465e0a1aa9f-kube-api-access-75c8r\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh\" (UID: \"d20b1e1e-b37d-451c-8540-0465e0a1aa9f\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh"
Sep 17 18:43:22.624755 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:22.624668    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh"
Sep 17 18:43:22.901386 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:22.901362    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh"]
Sep 17 18:43:22.903916 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:43:22.903893    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd20b1e1e_b37d_451c_8540_0465e0a1aa9f.slice/crio-7186f84a4f2331c8db95b1f39ca1812aa007a6ea278fd8b872020e644a9949d7 WatchSource:0}: Error finding container 7186f84a4f2331c8db95b1f39ca1812aa007a6ea278fd8b872020e644a9949d7: Status 404 returned error can't find the container with id 7186f84a4f2331c8db95b1f39ca1812aa007a6ea278fd8b872020e644a9949d7
Sep 17 18:43:23.841252 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:23.841216    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh" event={"ID":"d20b1e1e-b37d-451c-8540-0465e0a1aa9f","Type":"ContainerStarted","Data":"7fc38f8b1de0f69ee3f5a445fd2ec1e181f03fe84762e79b8baf276af6164a15"}
Sep 17 18:43:23.841714 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:23.841269    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh" event={"ID":"d20b1e1e-b37d-451c-8540-0465e0a1aa9f","Type":"ContainerStarted","Data":"7186f84a4f2331c8db95b1f39ca1812aa007a6ea278fd8b872020e644a9949d7"}
Sep 17 18:43:23.841714 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:23.841320    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh"
Sep 17 18:43:23.861466 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:23.861417    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh" podStartSLOduration=1.861401301 podStartE2EDuration="1.861401301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:43:23.860249479 +0000 UTC m=+840.711254878" watchObservedRunningTime="2026-09-17 18:43:23.861401301 +0000 UTC m=+840.712406722"
Sep 17 18:43:24.285263 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:24.285185    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:24.546796 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:24.546710    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:30.092721 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:30.092687    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:30.188184 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:30.188156    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:30.415693 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:30.415604    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:30.522427 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:30.522395    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:34.926447 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:34.926414    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:41.854500 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:41.854465    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:42.034100 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:42.034068    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29"]
Sep 17 18:43:42.037018 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:42.037001    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29"
Sep 17 18:43:42.040242 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:42.040203    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 17 18:43:42.040242 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:42.040234    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 17 18:43:42.040436 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:42.040254    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-fv6gz\""
Sep 17 18:43:42.040436 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:42.040322    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-p8j49\""
Sep 17 18:43:42.040567 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:42.040512    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 17 18:43:42.049909 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:42.049883    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29"]
Sep 17 18:43:42.060338 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:42.060312    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d10303e-8616-4d66-a26b-5c9c24450283-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29\" (UID: \"8d10303e-8616-4d66-a26b-5c9c24450283\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29"
Sep 17 18:43:42.060453 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:42.060350    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d10303e-8616-4d66-a26b-5c9c24450283-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29\" (UID: \"8d10303e-8616-4d66-a26b-5c9c24450283\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29"
Sep 17 18:43:42.060453 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:42.060377    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnmmd\" (UniqueName: \"kubernetes.io/projected/8d10303e-8616-4d66-a26b-5c9c24450283-kube-api-access-rnmmd\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29\" (UID: \"8d10303e-8616-4d66-a26b-5c9c24450283\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29"
Sep 17 18:43:42.060453 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:42.060424    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d10303e-8616-4d66-a26b-5c9c24450283-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29\" (UID: \"8d10303e-8616-4d66-a26b-5c9c24450283\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29"
Sep 17 18:43:42.075160 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:42.075128    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:42.160923 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:42.160838    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d10303e-8616-4d66-a26b-5c9c24450283-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29\" (UID: \"8d10303e-8616-4d66-a26b-5c9c24450283\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29"
Sep 17 18:43:42.160923 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:42.160894    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d10303e-8616-4d66-a26b-5c9c24450283-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29\" (UID: \"8d10303e-8616-4d66-a26b-5c9c24450283\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29"
Sep 17 18:43:42.160923 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:42.160924    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d10303e-8616-4d66-a26b-5c9c24450283-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29\" (UID: \"8d10303e-8616-4d66-a26b-5c9c24450283\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29"
Sep 17 18:43:42.161208 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:42.160945    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnmmd\" (UniqueName: \"kubernetes.io/projected/8d10303e-8616-4d66-a26b-5c9c24450283-kube-api-access-rnmmd\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29\" (UID: \"8d10303e-8616-4d66-a26b-5c9c24450283\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29"
Sep 17 18:43:42.161314 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:42.161293    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d10303e-8616-4d66-a26b-5c9c24450283-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29\" (UID: \"8d10303e-8616-4d66-a26b-5c9c24450283\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29"
Sep 17 18:43:42.161379 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:42.161310    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d10303e-8616-4d66-a26b-5c9c24450283-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29\" (UID: \"8d10303e-8616-4d66-a26b-5c9c24450283\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29"
Sep 17 18:43:42.163306 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:42.163288    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d10303e-8616-4d66-a26b-5c9c24450283-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29\" (UID: \"8d10303e-8616-4d66-a26b-5c9c24450283\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29"
Sep 17 18:43:42.169416 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:42.169371    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnmmd\" (UniqueName: \"kubernetes.io/projected/8d10303e-8616-4d66-a26b-5c9c24450283-kube-api-access-rnmmd\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29\" (UID: \"8d10303e-8616-4d66-a26b-5c9c24450283\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29"
Sep 17 18:43:42.350269 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:42.350233    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29"
Sep 17 18:43:42.624844 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:42.624808    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29"]
Sep 17 18:43:42.628139 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:43:42.628114    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d10303e_8616_4d66_a26b_5c9c24450283.slice/crio-8b6e6f73fbd378ee540388caadb779059401d9124480b41836898b189d8f09f1 WatchSource:0}: Error finding container 8b6e6f73fbd378ee540388caadb779059401d9124480b41836898b189d8f09f1: Status 404 returned error can't find the container with id 8b6e6f73fbd378ee540388caadb779059401d9124480b41836898b189d8f09f1
Sep 17 18:43:42.920520 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:42.920418    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29" event={"ID":"8d10303e-8616-4d66-a26b-5c9c24450283","Type":"ContainerStarted","Data":"a7b6638d6de051dfb04fc19c5583c403ad6883554a51f46f8de23a463fb1d04b"}
Sep 17 18:43:42.920520 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:42.920471    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29" event={"ID":"8d10303e-8616-4d66-a26b-5c9c24450283","Type":"ContainerStarted","Data":"8b6e6f73fbd378ee540388caadb779059401d9124480b41836898b189d8f09f1"}
Sep 17 18:43:42.921060 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:42.920513    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29"
Sep 17 18:43:42.940354 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:42.940300    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29" podStartSLOduration=0.940271873 podStartE2EDuration="940.271873ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:43:42.940222435 +0000 UTC m=+859.791227836" watchObservedRunningTime="2026-09-17 18:43:42.940271873 +0000 UTC m=+859.791277260"
Sep 17 18:43:49.487631 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:49.487588    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:54.847794 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:43:54.847765    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh"
Sep 17 18:44:00.901837 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:00.901805    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh"]
Sep 17 18:44:00.907636 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:00.907607    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:00.984119 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:00.984090    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:04.848467 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:04.848426    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh" podUID="d20b1e1e-b37d-451c-8540-0465e0a1aa9f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:44:13.926389 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:13.926315    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29"
Sep 17 18:44:14.551168 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:14.551136    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:14.652886 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:14.652854    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:14.847783 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:14.847697    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh" podUID="d20b1e1e-b37d-451c-8540-0465e0a1aa9f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:44:15.567616 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:15.567583    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29"]
Sep 17 18:44:15.581359 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:15.581329    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:15.670043 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:15.670011    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:15.921174 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:15.921061    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh" podUID="d20b1e1e-b37d-451c-8540-0465e0a1aa9f" containerName="main" containerID="cri-o://7fc38f8b1de0f69ee3f5a445fd2ec1e181f03fe84762e79b8baf276af6164a15" gracePeriod=45
Sep 17 18:44:23.925708 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:23.925673    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29" podUID="8d10303e-8616-4d66-a26b-5c9c24450283" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:44:24.847998 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:24.847967    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh" podUID="d20b1e1e-b37d-451c-8540-0465e0a1aa9f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:44:24.848165 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:24.848072    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh"
Sep 17 18:44:25.491652 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:25.491624    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xlklb_07ee31e0-43bb-4940-837d-969bd840492b/console-operator/1.log"
Sep 17 18:44:25.493604 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:25.493582    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xlklb_07ee31e0-43bb-4940-837d-969bd840492b/console-operator/1.log"
Sep 17 18:44:25.497070 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:25.497051    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovn-acl-logging/0.log"
Sep 17 18:44:25.498663 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:25.498646    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovn-acl-logging/0.log"
Sep 17 18:44:30.587595 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:30.587553    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29" podUID="8d10303e-8616-4d66-a26b-5c9c24450283" containerName="main" containerID="cri-o://a7b6638d6de051dfb04fc19c5583c403ad6883554a51f46f8de23a463fb1d04b" gracePeriod=45
Sep 17 18:44:33.926308 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:33.926273    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29" podUID="8d10303e-8616-4d66-a26b-5c9c24450283" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:44:34.847893 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:34.847855    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh" podUID="d20b1e1e-b37d-451c-8540-0465e0a1aa9f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:44:43.926199 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:43.926160    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29" podUID="8d10303e-8616-4d66-a26b-5c9c24450283" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:44:43.926597 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:43.926265    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29"
Sep 17 18:44:44.848418 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:44.848384    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh" podUID="d20b1e1e-b37d-451c-8540-0465e0a1aa9f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:44:46.160884 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:46.160857    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh"
Sep 17 18:44:46.172607 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:46.172579    2588 generic.go:356] "Generic (PLEG): container finished" podID="d20b1e1e-b37d-451c-8540-0465e0a1aa9f" containerID="7fc38f8b1de0f69ee3f5a445fd2ec1e181f03fe84762e79b8baf276af6164a15" exitCode=0
Sep 17 18:44:46.172730 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:46.172615    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh"
Sep 17 18:44:46.172730 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:46.172635    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh" event={"ID":"d20b1e1e-b37d-451c-8540-0465e0a1aa9f","Type":"ContainerDied","Data":"7fc38f8b1de0f69ee3f5a445fd2ec1e181f03fe84762e79b8baf276af6164a15"}
Sep 17 18:44:46.172730 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:46.172687    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh" event={"ID":"d20b1e1e-b37d-451c-8540-0465e0a1aa9f","Type":"ContainerDied","Data":"7186f84a4f2331c8db95b1f39ca1812aa007a6ea278fd8b872020e644a9949d7"}
Sep 17 18:44:46.172730 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:46.172718    2588 scope.go:117] "RemoveContainer" containerID="7fc38f8b1de0f69ee3f5a445fd2ec1e181f03fe84762e79b8baf276af6164a15"
Sep 17 18:44:46.175884 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:46.175859    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20b1e1e-b37d-451c-8540-0465e0a1aa9f-tls-certs\") pod \"d20b1e1e-b37d-451c-8540-0465e0a1aa9f\" (UID: \"d20b1e1e-b37d-451c-8540-0465e0a1aa9f\") "
Sep 17 18:44:46.175997 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:46.175943    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75c8r\" (UniqueName: \"kubernetes.io/projected/d20b1e1e-b37d-451c-8540-0465e0a1aa9f-kube-api-access-75c8r\") pod \"d20b1e1e-b37d-451c-8540-0465e0a1aa9f\" (UID: \"d20b1e1e-b37d-451c-8540-0465e0a1aa9f\") "
Sep 17 18:44:46.179136 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:46.178655    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d20b1e1e-b37d-451c-8540-0465e0a1aa9f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d20b1e1e-b37d-451c-8540-0465e0a1aa9f" (UID: "d20b1e1e-b37d-451c-8540-0465e0a1aa9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:44:46.179405 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:46.179377    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d20b1e1e-b37d-451c-8540-0465e0a1aa9f-kube-api-access-75c8r" (OuterVolumeSpecName: "kube-api-access-75c8r") pod "d20b1e1e-b37d-451c-8540-0465e0a1aa9f" (UID: "d20b1e1e-b37d-451c-8540-0465e0a1aa9f"). InnerVolumeSpecName "kube-api-access-75c8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:44:46.182811 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:46.182778    2588 scope.go:117] "RemoveContainer" containerID="7fc38f8b1de0f69ee3f5a445fd2ec1e181f03fe84762e79b8baf276af6164a15"
Sep 17 18:44:46.183134 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:44:46.183108    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fc38f8b1de0f69ee3f5a445fd2ec1e181f03fe84762e79b8baf276af6164a15\": container with ID starting with 7fc38f8b1de0f69ee3f5a445fd2ec1e181f03fe84762e79b8baf276af6164a15 not found: ID does not exist" containerID="7fc38f8b1de0f69ee3f5a445fd2ec1e181f03fe84762e79b8baf276af6164a15"
Sep 17 18:44:46.183219 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:46.183143    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fc38f8b1de0f69ee3f5a445fd2ec1e181f03fe84762e79b8baf276af6164a15"} err="failed to get container status \"7fc38f8b1de0f69ee3f5a445fd2ec1e181f03fe84762e79b8baf276af6164a15\": rpc error: code = NotFound desc = could not find container \"7fc38f8b1de0f69ee3f5a445fd2ec1e181f03fe84762e79b8baf276af6164a15\": container with ID starting with 7fc38f8b1de0f69ee3f5a445fd2ec1e181f03fe84762e79b8baf276af6164a15 not found: ID does not exist"
Sep 17 18:44:46.276954 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:46.276920    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75c8r\" (UniqueName: \"kubernetes.io/projected/d20b1e1e-b37d-451c-8540-0465e0a1aa9f-kube-api-access-75c8r\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:44:46.276954 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:46.276951    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20b1e1e-b37d-451c-8540-0465e0a1aa9f-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:44:46.493918 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:46.493884    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh"]
Sep 17 18:44:46.498281 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:46.498257    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-rs7vh"]
Sep 17 18:44:47.506138 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:47.506097    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d20b1e1e-b37d-451c-8540-0465e0a1aa9f" path="/var/lib/kubelet/pods/d20b1e1e-b37d-451c-8540-0465e0a1aa9f/volumes"
Sep 17 18:44:51.020589 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:51.020553    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:51.143791 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:51.143754    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:51.420506 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:51.420470    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:51.584269 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:51.584237    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:53.926262 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:53.926221    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29" podUID="8d10303e-8616-4d66-a26b-5c9c24450283" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:44:54.926542 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:54.926497    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:57.016413 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:44:57.016382    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:00.836297 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:00.836271    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29"
Sep 17 18:45:00.889477 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:00.889404    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d10303e-8616-4d66-a26b-5c9c24450283-tls-certs\") pod \"8d10303e-8616-4d66-a26b-5c9c24450283\" (UID: \"8d10303e-8616-4d66-a26b-5c9c24450283\") "
Sep 17 18:45:00.889477 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:00.889433    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d10303e-8616-4d66-a26b-5c9c24450283-tokenizer-cache\") pod \"8d10303e-8616-4d66-a26b-5c9c24450283\" (UID: \"8d10303e-8616-4d66-a26b-5c9c24450283\") "
Sep 17 18:45:00.889477 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:00.889452    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d10303e-8616-4d66-a26b-5c9c24450283-cachi2-cache\") pod \"8d10303e-8616-4d66-a26b-5c9c24450283\" (UID: \"8d10303e-8616-4d66-a26b-5c9c24450283\") "
Sep 17 18:45:00.889793 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:00.889483    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnmmd\" (UniqueName: \"kubernetes.io/projected/8d10303e-8616-4d66-a26b-5c9c24450283-kube-api-access-rnmmd\") pod \"8d10303e-8616-4d66-a26b-5c9c24450283\" (UID: \"8d10303e-8616-4d66-a26b-5c9c24450283\") "
Sep 17 18:45:00.889793 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:00.889733    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d10303e-8616-4d66-a26b-5c9c24450283-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "8d10303e-8616-4d66-a26b-5c9c24450283" (UID: "8d10303e-8616-4d66-a26b-5c9c24450283"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:00.889793 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:00.889748    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d10303e-8616-4d66-a26b-5c9c24450283-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "8d10303e-8616-4d66-a26b-5c9c24450283" (UID: "8d10303e-8616-4d66-a26b-5c9c24450283"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:00.891810 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:00.891782    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d10303e-8616-4d66-a26b-5c9c24450283-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8d10303e-8616-4d66-a26b-5c9c24450283" (UID: "8d10303e-8616-4d66-a26b-5c9c24450283"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:45:00.891895 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:00.891849    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d10303e-8616-4d66-a26b-5c9c24450283-kube-api-access-rnmmd" (OuterVolumeSpecName: "kube-api-access-rnmmd") pod "8d10303e-8616-4d66-a26b-5c9c24450283" (UID: "8d10303e-8616-4d66-a26b-5c9c24450283"). InnerVolumeSpecName "kube-api-access-rnmmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:45:00.990188 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:00.990158    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d10303e-8616-4d66-a26b-5c9c24450283-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:45:00.990188 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:00.990183    2588 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d10303e-8616-4d66-a26b-5c9c24450283-tokenizer-cache\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:45:00.990188 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:00.990193    2588 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d10303e-8616-4d66-a26b-5c9c24450283-cachi2-cache\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:45:00.990411 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:00.990202    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnmmd\" (UniqueName: \"kubernetes.io/projected/8d10303e-8616-4d66-a26b-5c9c24450283-kube-api-access-rnmmd\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:45:01.235455 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:01.235360    2588 generic.go:356] "Generic (PLEG): container finished" podID="8d10303e-8616-4d66-a26b-5c9c24450283" containerID="a7b6638d6de051dfb04fc19c5583c403ad6883554a51f46f8de23a463fb1d04b" exitCode=0
Sep 17 18:45:01.235455 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:01.235407    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29" event={"ID":"8d10303e-8616-4d66-a26b-5c9c24450283","Type":"ContainerDied","Data":"a7b6638d6de051dfb04fc19c5583c403ad6883554a51f46f8de23a463fb1d04b"}
Sep 17 18:45:01.235455 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:01.235411    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29"
Sep 17 18:45:01.235455 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:01.235450    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29" event={"ID":"8d10303e-8616-4d66-a26b-5c9c24450283","Type":"ContainerDied","Data":"8b6e6f73fbd378ee540388caadb779059401d9124480b41836898b189d8f09f1"}
Sep 17 18:45:01.235820 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:01.235473    2588 scope.go:117] "RemoveContainer" containerID="a7b6638d6de051dfb04fc19c5583c403ad6883554a51f46f8de23a463fb1d04b"
Sep 17 18:45:01.244372 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:01.244354    2588 scope.go:117] "RemoveContainer" containerID="a7b6638d6de051dfb04fc19c5583c403ad6883554a51f46f8de23a463fb1d04b"
Sep 17 18:45:01.244710 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:45:01.244690    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7b6638d6de051dfb04fc19c5583c403ad6883554a51f46f8de23a463fb1d04b\": container with ID starting with a7b6638d6de051dfb04fc19c5583c403ad6883554a51f46f8de23a463fb1d04b not found: ID does not exist" containerID="a7b6638d6de051dfb04fc19c5583c403ad6883554a51f46f8de23a463fb1d04b"
Sep 17 18:45:01.244780 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:01.244719    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7b6638d6de051dfb04fc19c5583c403ad6883554a51f46f8de23a463fb1d04b"} err="failed to get container status \"a7b6638d6de051dfb04fc19c5583c403ad6883554a51f46f8de23a463fb1d04b\": rpc error: code = NotFound desc = could not find container \"a7b6638d6de051dfb04fc19c5583c403ad6883554a51f46f8de23a463fb1d04b\": container with ID starting with a7b6638d6de051dfb04fc19c5583c403ad6883554a51f46f8de23a463fb1d04b not found: ID does not exist"
Sep 17 18:45:01.256654 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:01.256630    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29"]
Sep 17 18:45:01.261657 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:01.261634    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcqxw29"]
Sep 17 18:45:01.506399 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:01.506314    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d10303e-8616-4d66-a26b-5c9c24450283" path="/var/lib/kubelet/pods/8d10303e-8616-4d66-a26b-5c9c24450283/volumes"
Sep 17 18:45:02.777264 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:02.777185    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:02.876663 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:02.876631    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:05.196717 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:05.196683    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:05.313936 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:05.313901    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:07.861882 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:07.861846    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:11.038860 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.038824    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:11.147643 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.147608    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:11.264520 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.264487    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:11.377573 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.377542    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"]
Sep 17 18:45:11.377984 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.377966    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d10303e-8616-4d66-a26b-5c9c24450283" containerName="main"
Sep 17 18:45:11.378032 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.377989    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d10303e-8616-4d66-a26b-5c9c24450283" containerName="main"
Sep 17 18:45:11.378032 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.378014    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d20b1e1e-b37d-451c-8540-0465e0a1aa9f" containerName="main"
Sep 17 18:45:11.378032 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.378023    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="d20b1e1e-b37d-451c-8540-0465e0a1aa9f" containerName="main"
Sep 17 18:45:11.378127 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.378114    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d10303e-8616-4d66-a26b-5c9c24450283" containerName="main"
Sep 17 18:45:11.378159 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.378126    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="d20b1e1e-b37d-451c-8540-0465e0a1aa9f" containerName="main"
Sep 17 18:45:11.387355 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.387323    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:45:11.389892 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.389871    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 17 18:45:11.390023 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.389890    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Sep 17 18:45:11.390023 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.389890    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-klnnd\""
Sep 17 18:45:11.390541 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.390514    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 17 18:45:11.402648 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.402627    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"]
Sep 17 18:45:11.451547 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.451505    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:11.486653 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.486623    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14a9404b-d884-4cf3-93c7-12e06c2309ec-istio-envoy\") pod \"router-gateway-1-openshift-default-7dcbd489bd-fgtw7\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:45:11.486819 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.486669    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14a9404b-d884-4cf3-93c7-12e06c2309ec-istio-podinfo\") pod \"router-gateway-1-openshift-default-7dcbd489bd-fgtw7\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:45:11.486819 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.486687    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75q95\" (UniqueName: \"kubernetes.io/projected/14a9404b-d884-4cf3-93c7-12e06c2309ec-kube-api-access-75q95\") pod \"router-gateway-1-openshift-default-7dcbd489bd-fgtw7\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:45:11.486819 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.486742    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a9404b-d884-4cf3-93c7-12e06c2309ec-workload-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-fgtw7\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:45:11.486819 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.486772    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a9404b-d884-4cf3-93c7-12e06c2309ec-credential-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-fgtw7\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:45:11.486819 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.486804    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14a9404b-d884-4cf3-93c7-12e06c2309ec-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7dcbd489bd-fgtw7\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:45:11.486819 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.486821    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14a9404b-d884-4cf3-93c7-12e06c2309ec-istio-token\") pod \"router-gateway-1-openshift-default-7dcbd489bd-fgtw7\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:45:11.487112 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.486891    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14a9404b-d884-4cf3-93c7-12e06c2309ec-workload-certs\") pod \"router-gateway-1-openshift-default-7dcbd489bd-fgtw7\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:45:11.487112 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.486929    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14a9404b-d884-4cf3-93c7-12e06c2309ec-istio-data\") pod \"router-gateway-1-openshift-default-7dcbd489bd-fgtw7\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:45:11.588051 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.588016    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14a9404b-d884-4cf3-93c7-12e06c2309ec-istio-envoy\") pod \"router-gateway-1-openshift-default-7dcbd489bd-fgtw7\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:45:11.588228 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.588067    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14a9404b-d884-4cf3-93c7-12e06c2309ec-istio-podinfo\") pod \"router-gateway-1-openshift-default-7dcbd489bd-fgtw7\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:45:11.588228 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.588095    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75q95\" (UniqueName: \"kubernetes.io/projected/14a9404b-d884-4cf3-93c7-12e06c2309ec-kube-api-access-75q95\") pod \"router-gateway-1-openshift-default-7dcbd489bd-fgtw7\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:45:11.588228 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.588119    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a9404b-d884-4cf3-93c7-12e06c2309ec-workload-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-fgtw7\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:45:11.588228 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.588143    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a9404b-d884-4cf3-93c7-12e06c2309ec-credential-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-fgtw7\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:45:11.588228 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.588179    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14a9404b-d884-4cf3-93c7-12e06c2309ec-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7dcbd489bd-fgtw7\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:45:11.588228 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.588202    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14a9404b-d884-4cf3-93c7-12e06c2309ec-istio-token\") pod \"router-gateway-1-openshift-default-7dcbd489bd-fgtw7\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:45:11.588572 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.588239    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14a9404b-d884-4cf3-93c7-12e06c2309ec-workload-certs\") pod \"router-gateway-1-openshift-default-7dcbd489bd-fgtw7\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:45:11.588572 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.588266    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14a9404b-d884-4cf3-93c7-12e06c2309ec-istio-data\") pod \"router-gateway-1-openshift-default-7dcbd489bd-fgtw7\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:45:11.588686 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.588608    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a9404b-d884-4cf3-93c7-12e06c2309ec-workload-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-fgtw7\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:45:11.588873 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.588853    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14a9404b-d884-4cf3-93c7-12e06c2309ec-istio-data\") pod \"router-gateway-1-openshift-default-7dcbd489bd-fgtw7\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:45:11.588920 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.588893    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a9404b-d884-4cf3-93c7-12e06c2309ec-credential-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-fgtw7\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:45:11.588954 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.588901    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14a9404b-d884-4cf3-93c7-12e06c2309ec-workload-certs\") pod \"router-gateway-1-openshift-default-7dcbd489bd-fgtw7\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:45:11.589128 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.589104    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14a9404b-d884-4cf3-93c7-12e06c2309ec-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7dcbd489bd-fgtw7\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:45:11.590420 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.590399    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14a9404b-d884-4cf3-93c7-12e06c2309ec-istio-envoy\") pod \"router-gateway-1-openshift-default-7dcbd489bd-fgtw7\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:45:11.590680 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.590661    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14a9404b-d884-4cf3-93c7-12e06c2309ec-istio-podinfo\") pod \"router-gateway-1-openshift-default-7dcbd489bd-fgtw7\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:45:11.603858 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.603830    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75q95\" (UniqueName: \"kubernetes.io/projected/14a9404b-d884-4cf3-93c7-12e06c2309ec-kube-api-access-75q95\") pod \"router-gateway-1-openshift-default-7dcbd489bd-fgtw7\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:45:11.603946 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.603882    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14a9404b-d884-4cf3-93c7-12e06c2309ec-istio-token\") pod \"router-gateway-1-openshift-default-7dcbd489bd-fgtw7\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:45:11.623777 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.623718    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:11.698306 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.698270    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:45:11.879391 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.879286    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:11.976134 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.976111    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"]
Sep 17 18:45:11.978597 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:45:11.978570    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a9404b_d884_4cf3_93c7_12e06c2309ec.slice/crio-a922533723f471797408b98cb9d1fa2630fff44e6c7fa0615c2f5520d27011e7 WatchSource:0}: Error finding container a922533723f471797408b98cb9d1fa2630fff44e6c7fa0615c2f5520d27011e7: Status 404 returned error can't find the container with id a922533723f471797408b98cb9d1fa2630fff44e6c7fa0615c2f5520d27011e7
Sep 17 18:45:11.980650 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.980618    2588 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 18:45:11.980740 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.980683    2588 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 18:45:11.980740 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:11.980711    2588 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 18:45:12.284220 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:12.284123    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7" event={"ID":"14a9404b-d884-4cf3-93c7-12e06c2309ec","Type":"ContainerStarted","Data":"370920681e4bc3657ba26f1fd20edf603576da6437bac7cd62d91f1839838891"}
Sep 17 18:45:12.284220 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:12.284189    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7" event={"ID":"14a9404b-d884-4cf3-93c7-12e06c2309ec","Type":"ContainerStarted","Data":"a922533723f471797408b98cb9d1fa2630fff44e6c7fa0615c2f5520d27011e7"}
Sep 17 18:45:12.304696 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:12.304645    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7" podStartSLOduration=1.304629809 podStartE2EDuration="1.304629809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:45:12.303397901 +0000 UTC m=+949.154403300" watchObservedRunningTime="2026-09-17 18:45:12.304629809 +0000 UTC m=+949.155635207"
Sep 17 18:45:12.698767 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:12.698726    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:45:13.703613 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:13.703581    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:45:14.209634 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:14.209595    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:14.295668 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:14.295638    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:45:14.296639 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:14.296615    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:45:16.111025 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:16.110988    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:17.682207 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:17.682177    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:17.765259 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:17.765232    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:22.317357 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:22.317316    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:22.418677 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:22.418632    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:23.429895 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:23.429846    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:27.631890 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:27.631846    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:27.733442 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:27.733400    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:28.585391 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:28.585354    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:35.756218 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:35.756178    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:51.895993 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:51.895950    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676"]
Sep 17 18:45:51.899644 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:51.899620    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676"
Sep 17 18:45:51.903824 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:51.903798    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-txtkm\""
Sep 17 18:45:51.903958 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:51.903913    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 17 18:45:51.904112 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:51.904089    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 17 18:45:51.904246 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:51.904161    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 17 18:45:51.919433 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:51.919410    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676"]
Sep 17 18:45:51.919818 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:51.919792    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676\" (UID: \"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676"
Sep 17 18:45:51.919873 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:51.919832    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676\" (UID: \"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676"
Sep 17 18:45:51.919873 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:51.919862    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szk5j\" (UniqueName: \"kubernetes.io/projected/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-kube-api-access-szk5j\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676\" (UID: \"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676"
Sep 17 18:45:51.919947 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:51.919891    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676\" (UID: \"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676"
Sep 17 18:45:51.919947 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:51.919916    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676\" (UID: \"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676"
Sep 17 18:45:51.920010 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:51.919980    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676\" (UID: \"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676"
Sep 17 18:45:51.949782 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:51.949745    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:52.020966 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:52.020930    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676\" (UID: \"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676"
Sep 17 18:45:52.020966 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:52.020966    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676\" (UID: \"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676"
Sep 17 18:45:52.021189 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:52.020986    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szk5j\" (UniqueName: \"kubernetes.io/projected/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-kube-api-access-szk5j\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676\" (UID: \"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676"
Sep 17 18:45:52.021189 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:52.021006    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676\" (UID: \"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676"
Sep 17 18:45:52.021189 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:52.021029    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676\" (UID: \"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676"
Sep 17 18:45:52.021189 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:52.021053    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676\" (UID: \"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676"
Sep 17 18:45:52.021438 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:52.021418    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676\" (UID: \"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676"
Sep 17 18:45:52.021500 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:52.021474    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676\" (UID: \"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676"
Sep 17 18:45:52.021589 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:52.021510    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676\" (UID: \"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676"
Sep 17 18:45:52.023316 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:52.023298    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676\" (UID: \"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676"
Sep 17 18:45:52.023496 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:52.023479    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676\" (UID: \"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676"
Sep 17 18:45:52.029681 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:52.029660    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szk5j\" (UniqueName: \"kubernetes.io/projected/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-kube-api-access-szk5j\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676\" (UID: \"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676"
Sep 17 18:45:52.154419 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:52.154342    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb"]
Sep 17 18:45:52.157821 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:52.157805    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb"
Sep 17 18:45:52.160925 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:52.160900    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-v4vv9\""
Sep 17 18:45:52.173677 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:52.173654    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb"]
Sep 17 18:45:52.206984 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:52.206952    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:52.212042 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:52.212024    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676"
Sep 17 18:45:52.223027 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:52.222997    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b09b566-c9cf-4020-8dac-7399d24f9739-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb\" (UID: \"0b09b566-c9cf-4020-8dac-7399d24f9739\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb"
Sep 17 18:45:52.223131 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:52.223044    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2lct\" (UniqueName: \"kubernetes.io/projected/0b09b566-c9cf-4020-8dac-7399d24f9739-kube-api-access-x2lct\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb\" (UID: \"0b09b566-c9cf-4020-8dac-7399d24f9739\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb"
Sep 17 18:45:52.324398 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:52.324365    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b09b566-c9cf-4020-8dac-7399d24f9739-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb\" (UID: \"0b09b566-c9cf-4020-8dac-7399d24f9739\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb"
Sep 17 18:45:52.324615 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:52.324412    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2lct\" (UniqueName: \"kubernetes.io/projected/0b09b566-c9cf-4020-8dac-7399d24f9739-kube-api-access-x2lct\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb\" (UID: \"0b09b566-c9cf-4020-8dac-7399d24f9739\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb"
Sep 17 18:45:52.326890 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:52.326870    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b09b566-c9cf-4020-8dac-7399d24f9739-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb\" (UID: \"0b09b566-c9cf-4020-8dac-7399d24f9739\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb"
Sep 17 18:45:52.333446 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:52.333422    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2lct\" (UniqueName: \"kubernetes.io/projected/0b09b566-c9cf-4020-8dac-7399d24f9739-kube-api-access-x2lct\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb\" (UID: \"0b09b566-c9cf-4020-8dac-7399d24f9739\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb"
Sep 17 18:45:52.469143 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:52.469048    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb"
Sep 17 18:45:52.499189 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:52.499154    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676"]
Sep 17 18:45:52.504009 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:45:52.503977    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ddabc2_b7c8_49b0_ad82_8f64a0fd1d58.slice/crio-b9d1e2619c974df4f423c3a092c2d498b3b2b3d3ea67eadf47c8915d97c57d27 WatchSource:0}: Error finding container b9d1e2619c974df4f423c3a092c2d498b3b2b3d3ea67eadf47c8915d97c57d27: Status 404 returned error can't find the container with id b9d1e2619c974df4f423c3a092c2d498b3b2b3d3ea67eadf47c8915d97c57d27
Sep 17 18:45:52.761545 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:52.761108    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb"]
Sep 17 18:45:52.766352 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:45:52.766320    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b09b566_c9cf_4020_8dac_7399d24f9739.slice/crio-162d894fb2997d3ecd25cfd426ec168f826e739926b6e72d973fc33b5ef7b3ef WatchSource:0}: Error finding container 162d894fb2997d3ecd25cfd426ec168f826e739926b6e72d973fc33b5ef7b3ef: Status 404 returned error can't find the container with id 162d894fb2997d3ecd25cfd426ec168f826e739926b6e72d973fc33b5ef7b3ef
Sep 17 18:45:53.451372 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:53.451304    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676" event={"ID":"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58","Type":"ContainerStarted","Data":"b9d1e2619c974df4f423c3a092c2d498b3b2b3d3ea67eadf47c8915d97c57d27"}
Sep 17 18:45:53.453170 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:53.453142    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb" event={"ID":"0b09b566-c9cf-4020-8dac-7399d24f9739","Type":"ContainerStarted","Data":"3c44cc98b3bee39ae34c6e79ef3d848aad186277de5de901d9bf4863199f666a"}
Sep 17 18:45:53.453295 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:53.453197    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb" event={"ID":"0b09b566-c9cf-4020-8dac-7399d24f9739","Type":"ContainerStarted","Data":"162d894fb2997d3ecd25cfd426ec168f826e739926b6e72d973fc33b5ef7b3ef"}
Sep 17 18:45:53.453295 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:53.453231    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb"
Sep 17 18:45:55.468992 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:55.468946    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676" event={"ID":"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58","Type":"ContainerStarted","Data":"6fb790f185094e61c5b1ae23b2329f8b2ea0cb9bc22acda7aa500c0a4631be8c"}
Sep 17 18:45:55.487740 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:55.487680    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb" podStartSLOduration=3.487663636 podStartE2EDuration="3.487663636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:45:53.518613758 +0000 UTC m=+990.369619158" watchObservedRunningTime="2026-09-17 18:45:55.487663636 +0000 UTC m=+992.338669034"
Sep 17 18:45:55.488020 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:55.487987    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676" podStartSLOduration=2.400952403 podStartE2EDuration="4.487976462s" totalImagesPullingTime="2.087024059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:45:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:45:55.485918897 +0000 UTC m=+992.336924297" watchObservedRunningTime="2026-09-17 18:45:55.487976462 +0000 UTC m=+992.338981859"
Sep 17 18:45:56.212801 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:56.212756    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676"
Sep 17 18:45:59.504983 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:59.504939    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:59.602649 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:45:59.602614    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:02.214465 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:02.214421    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676"
Sep 17 18:46:02.234240 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:02.234214    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676"
Sep 17 18:46:02.527216 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:02.526835    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676"
Sep 17 18:46:02.538642 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:02.538611    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:04.036872 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:04.036837    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:07.507683 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:07.507653    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"]
Sep 17 18:46:07.508251 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:07.507893    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7" podUID="14a9404b-d884-4cf3-93c7-12e06c2309ec" containerName="istio-proxy" containerID="cri-o://370920681e4bc3657ba26f1fd20edf603576da6437bac7cd62d91f1839838891" gracePeriod=30
Sep 17 18:46:07.574591 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:07.574548    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:09.542717 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:09.542691    2588 generic.go:356] "Generic (PLEG): container finished" podID="14a9404b-d884-4cf3-93c7-12e06c2309ec" containerID="370920681e4bc3657ba26f1fd20edf603576da6437bac7cd62d91f1839838891" exitCode=0
Sep 17 18:46:09.543068 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:09.542763    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7" event={"ID":"14a9404b-d884-4cf3-93c7-12e06c2309ec","Type":"ContainerDied","Data":"370920681e4bc3657ba26f1fd20edf603576da6437bac7cd62d91f1839838891"}
Sep 17 18:46:09.543068 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:09.542830    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7" event={"ID":"14a9404b-d884-4cf3-93c7-12e06c2309ec","Type":"ContainerDied","Data":"a922533723f471797408b98cb9d1fa2630fff44e6c7fa0615c2f5520d27011e7"}
Sep 17 18:46:09.543068 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:09.542855    2588 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a922533723f471797408b98cb9d1fa2630fff44e6c7fa0615c2f5520d27011e7"
Sep 17 18:46:09.545769 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:09.545745    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:46:09.685495 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:09.685461    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14a9404b-d884-4cf3-93c7-12e06c2309ec-workload-certs\") pod \"14a9404b-d884-4cf3-93c7-12e06c2309ec\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") "
Sep 17 18:46:09.685495 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:09.685495    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a9404b-d884-4cf3-93c7-12e06c2309ec-workload-socket\") pod \"14a9404b-d884-4cf3-93c7-12e06c2309ec\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") "
Sep 17 18:46:09.685742 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:09.685580    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14a9404b-d884-4cf3-93c7-12e06c2309ec-istio-envoy\") pod \"14a9404b-d884-4cf3-93c7-12e06c2309ec\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") "
Sep 17 18:46:09.685742 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:09.685602    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75q95\" (UniqueName: \"kubernetes.io/projected/14a9404b-d884-4cf3-93c7-12e06c2309ec-kube-api-access-75q95\") pod \"14a9404b-d884-4cf3-93c7-12e06c2309ec\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") "
Sep 17 18:46:09.685742 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:09.685626    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14a9404b-d884-4cf3-93c7-12e06c2309ec-istio-podinfo\") pod \"14a9404b-d884-4cf3-93c7-12e06c2309ec\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") "
Sep 17 18:46:09.685742 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:09.685648    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14a9404b-d884-4cf3-93c7-12e06c2309ec-istio-data\") pod \"14a9404b-d884-4cf3-93c7-12e06c2309ec\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") "
Sep 17 18:46:09.685742 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:09.685699    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a9404b-d884-4cf3-93c7-12e06c2309ec-credential-socket\") pod \"14a9404b-d884-4cf3-93c7-12e06c2309ec\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") "
Sep 17 18:46:09.685742 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:09.685739    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14a9404b-d884-4cf3-93c7-12e06c2309ec-istio-token\") pod \"14a9404b-d884-4cf3-93c7-12e06c2309ec\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") "
Sep 17 18:46:09.686059 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:09.685786    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14a9404b-d884-4cf3-93c7-12e06c2309ec-istiod-ca-cert\") pod \"14a9404b-d884-4cf3-93c7-12e06c2309ec\" (UID: \"14a9404b-d884-4cf3-93c7-12e06c2309ec\") "
Sep 17 18:46:09.686059 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:09.685829    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a9404b-d884-4cf3-93c7-12e06c2309ec-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "14a9404b-d884-4cf3-93c7-12e06c2309ec" (UID: "14a9404b-d884-4cf3-93c7-12e06c2309ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:46:09.686059 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:09.685919    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a9404b-d884-4cf3-93c7-12e06c2309ec-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "14a9404b-d884-4cf3-93c7-12e06c2309ec" (UID: "14a9404b-d884-4cf3-93c7-12e06c2309ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:46:09.686059 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:09.686009    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a9404b-d884-4cf3-93c7-12e06c2309ec-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "14a9404b-d884-4cf3-93c7-12e06c2309ec" (UID: "14a9404b-d884-4cf3-93c7-12e06c2309ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:46:09.686282 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:09.686128    2588 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a9404b-d884-4cf3-93c7-12e06c2309ec-credential-socket\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:46:09.686282 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:09.686149    2588 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14a9404b-d884-4cf3-93c7-12e06c2309ec-workload-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:46:09.686282 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:09.686164    2588 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14a9404b-d884-4cf3-93c7-12e06c2309ec-workload-socket\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:46:09.686282 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:09.686165    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a9404b-d884-4cf3-93c7-12e06c2309ec-istio-data" (OuterVolumeSpecName: "istio-data") pod "14a9404b-d884-4cf3-93c7-12e06c2309ec" (UID: "14a9404b-d884-4cf3-93c7-12e06c2309ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:46:09.686282 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:09.686228    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a9404b-d884-4cf3-93c7-12e06c2309ec-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "14a9404b-d884-4cf3-93c7-12e06c2309ec" (UID: "14a9404b-d884-4cf3-93c7-12e06c2309ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:46:09.688063 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:09.688041    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14a9404b-d884-4cf3-93c7-12e06c2309ec-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "14a9404b-d884-4cf3-93c7-12e06c2309ec" (UID: "14a9404b-d884-4cf3-93c7-12e06c2309ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:46:09.688063 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:09.688047    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a9404b-d884-4cf3-93c7-12e06c2309ec-istio-token" (OuterVolumeSpecName: "istio-token") pod "14a9404b-d884-4cf3-93c7-12e06c2309ec" (UID: "14a9404b-d884-4cf3-93c7-12e06c2309ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:46:09.688316 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:09.688294    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14a9404b-d884-4cf3-93c7-12e06c2309ec-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "14a9404b-d884-4cf3-93c7-12e06c2309ec" (UID: "14a9404b-d884-4cf3-93c7-12e06c2309ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 18:46:09.688405 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:09.688319    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14a9404b-d884-4cf3-93c7-12e06c2309ec-kube-api-access-75q95" (OuterVolumeSpecName: "kube-api-access-75q95") pod "14a9404b-d884-4cf3-93c7-12e06c2309ec" (UID: "14a9404b-d884-4cf3-93c7-12e06c2309ec"). InnerVolumeSpecName "kube-api-access-75q95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:46:09.786882 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:09.786847    2588 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14a9404b-d884-4cf3-93c7-12e06c2309ec-istiod-ca-cert\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:46:09.786882 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:09.786872    2588 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14a9404b-d884-4cf3-93c7-12e06c2309ec-istio-envoy\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:46:09.786882 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:09.786881    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75q95\" (UniqueName: \"kubernetes.io/projected/14a9404b-d884-4cf3-93c7-12e06c2309ec-kube-api-access-75q95\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:46:09.786882 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:09.786890    2588 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14a9404b-d884-4cf3-93c7-12e06c2309ec-istio-podinfo\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:46:09.787143 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:09.786898    2588 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14a9404b-d884-4cf3-93c7-12e06c2309ec-istio-data\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:46:09.787143 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:09.786908    2588 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14a9404b-d884-4cf3-93c7-12e06c2309ec-istio-token\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:46:10.547561 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:10.547517    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"
Sep 17 18:46:10.569392 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:10.569364    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"]
Sep 17 18:46:10.574854 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:10.574829    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-fgtw7"]
Sep 17 18:46:11.506370 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:11.506336    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14a9404b-d884-4cf3-93c7-12e06c2309ec" path="/var/lib/kubelet/pods/14a9404b-d884-4cf3-93c7-12e06c2309ec/volumes"
Sep 17 18:46:12.910396 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:12.910364    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:13.010107 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:13.010070    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:13.938747 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:13.938708    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:14.315845 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:14.315814    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:19.928336 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:19.928301    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:24.092432 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:24.092399    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:24.462192 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:24.462165    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb"
Sep 17 18:46:26.099508 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:26.099473    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:26.252192 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:26.252158    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:32.501713 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:32.501677    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:32.786156 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:32.786072    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx"]
Sep 17 18:46:32.786441 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:32.786424    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14a9404b-d884-4cf3-93c7-12e06c2309ec" containerName="istio-proxy"
Sep 17 18:46:32.786522 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:32.786445    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="14a9404b-d884-4cf3-93c7-12e06c2309ec" containerName="istio-proxy"
Sep 17 18:46:32.786611 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:32.786558    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="14a9404b-d884-4cf3-93c7-12e06c2309ec" containerName="istio-proxy"
Sep 17 18:46:32.790673 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:32.790653    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx"
Sep 17 18:46:32.793899 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:32.793879    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-8h246\""
Sep 17 18:46:32.794068 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:32.794048    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 17 18:46:32.794118 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:32.794076    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 17 18:46:32.794165 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:32.794120    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 17 18:46:32.794165 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:32.794143    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-nt62k\""
Sep 17 18:46:32.802194 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:32.802175    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx"]
Sep 17 18:46:32.827962 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:32.827939    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:32.884828 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:32.884799    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8fd6\" (UniqueName: \"kubernetes.io/projected/2757fb04-a4af-4f0b-9dbd-ff2fa4624adb-kube-api-access-l8fd6\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx\" (UID: \"2757fb04-a4af-4f0b-9dbd-ff2fa4624adb\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx"
Sep 17 18:46:32.884962 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:32.884904    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2757fb04-a4af-4f0b-9dbd-ff2fa4624adb-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx\" (UID: \"2757fb04-a4af-4f0b-9dbd-ff2fa4624adb\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx"
Sep 17 18:46:32.985885 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:32.985846    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2757fb04-a4af-4f0b-9dbd-ff2fa4624adb-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx\" (UID: \"2757fb04-a4af-4f0b-9dbd-ff2fa4624adb\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx"
Sep 17 18:46:32.986065 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:32.985892    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8fd6\" (UniqueName: \"kubernetes.io/projected/2757fb04-a4af-4f0b-9dbd-ff2fa4624adb-kube-api-access-l8fd6\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx\" (UID: \"2757fb04-a4af-4f0b-9dbd-ff2fa4624adb\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx"
Sep 17 18:46:32.988451 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:32.988413    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2757fb04-a4af-4f0b-9dbd-ff2fa4624adb-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx\" (UID: \"2757fb04-a4af-4f0b-9dbd-ff2fa4624adb\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx"
Sep 17 18:46:32.996495 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:32.996468    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8fd6\" (UniqueName: \"kubernetes.io/projected/2757fb04-a4af-4f0b-9dbd-ff2fa4624adb-kube-api-access-l8fd6\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx\" (UID: \"2757fb04-a4af-4f0b-9dbd-ff2fa4624adb\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx"
Sep 17 18:46:33.101636 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:33.101598    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx"
Sep 17 18:46:33.388155 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:33.388059    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx"]
Sep 17 18:46:33.392086 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:46:33.392057    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2757fb04_a4af_4f0b_9dbd_ff2fa4624adb.slice/crio-71a8056c44295af9b05924956463dac5b4fe2e37b3722919982a5480614d73c6 WatchSource:0}: Error finding container 71a8056c44295af9b05924956463dac5b4fe2e37b3722919982a5480614d73c6: Status 404 returned error can't find the container with id 71a8056c44295af9b05924956463dac5b4fe2e37b3722919982a5480614d73c6
Sep 17 18:46:33.653481 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:33.653399    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx" event={"ID":"2757fb04-a4af-4f0b-9dbd-ff2fa4624adb","Type":"ContainerStarted","Data":"e84c6cfd94ddac388a94ecee6ffcaedef323c26cfaa1cb1f39df989797c56dbc"}
Sep 17 18:46:33.653481 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:33.653462    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx" event={"ID":"2757fb04-a4af-4f0b-9dbd-ff2fa4624adb","Type":"ContainerStarted","Data":"71a8056c44295af9b05924956463dac5b4fe2e37b3722919982a5480614d73c6"}
Sep 17 18:46:33.653918 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:33.653627    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx"
Sep 17 18:46:33.671985 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:33.671934    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx" podStartSLOduration=1.6719162239999998 podStartE2EDuration="1.671916224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:46:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:46:33.671361678 +0000 UTC m=+1030.522367079" watchObservedRunningTime="2026-09-17 18:46:33.671916224 +0000 UTC m=+1030.522921624"
Sep 17 18:46:50.227710 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:46:50.227667    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:02.053387 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:02.053352    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:04.659619 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:04.659590    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx"
Sep 17 18:47:20.060922 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:20.060838    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:20.149519 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:20.149488    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:32.340901 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:32.340861    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:43.967578 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:43.967544    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:43.986161 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:43.986122    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676"]
Sep 17 18:47:43.987780 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:43.987747    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb"]
Sep 17 18:47:43.995891 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:47:43.995849    2588 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:47:43.995891 ip-10-0-136-132 kubenswrapper[2588]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:47:43.995891 ip-10-0-136-132 kubenswrapper[2588]:         , stdout: , stderr: , exit code -1
Sep 17 18:47:43.995891 ip-10-0-136-132 kubenswrapper[2588]:  > containerID="6fb790f185094e61c5b1ae23b2329f8b2ea0cb9bc22acda7aa500c0a4631be8c" cmd=["/bin/sleep","15"]
Sep 17 18:47:43.996124 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:43.995906    2588 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676" message="" err=<
Sep 17 18:47:43.996124 ip-10-0-136-132 kubenswrapper[2588]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:47:43.996124 ip-10-0-136-132 kubenswrapper[2588]:         , stdout: , stderr: , exit code -1
Sep 17 18:47:43.996124 ip-10-0-136-132 kubenswrapper[2588]:  >
Sep 17 18:47:43.996124 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:47:43.995919    2588 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:47:43.996124 ip-10-0-136-132 kubenswrapper[2588]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:47:43.996124 ip-10-0-136-132 kubenswrapper[2588]:         , stdout: , stderr: , exit code -1
Sep 17 18:47:43.996124 ip-10-0-136-132 kubenswrapper[2588]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676" podUID="c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58" containerName="main" containerID="cri-o://6fb790f185094e61c5b1ae23b2329f8b2ea0cb9bc22acda7aa500c0a4631be8c"
Sep 17 18:47:43.996124 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:43.995949    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676" podUID="c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58" containerName="main" containerID="cri-o://6fb790f185094e61c5b1ae23b2329f8b2ea0cb9bc22acda7aa500c0a4631be8c" gracePeriod=60
Sep 17 18:47:44.050356 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:44.050325    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:47:44.235610 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:44.235582    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676"
Sep 17 18:47:44.289804 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:44.289777    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szk5j\" (UniqueName: \"kubernetes.io/projected/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-kube-api-access-szk5j\") pod \"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58\" (UID: \"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58\") "
Sep 17 18:47:44.289943 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:44.289812    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-model-cache\") pod \"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58\" (UID: \"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58\") "
Sep 17 18:47:44.289943 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:44.289842    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-tls-certs\") pod \"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58\" (UID: \"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58\") "
Sep 17 18:47:44.289943 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:44.289894    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-dshm\") pod \"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58\" (UID: \"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58\") "
Sep 17 18:47:44.289943 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:44.289926    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-tmp-dir\") pod \"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58\" (UID: \"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58\") "
Sep 17 18:47:44.290156 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:44.289948    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-home\") pod \"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58\" (UID: \"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58\") "
Sep 17 18:47:44.290156 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:44.290016    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-model-cache" (OuterVolumeSpecName: "model-cache") pod "c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58" (UID: "c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:47:44.290268 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:44.290167    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58" (UID: "c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:47:44.290268 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:44.290237    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-home" (OuterVolumeSpecName: "home") pod "c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58" (UID: "c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:47:44.290268 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:44.290247    2588 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-model-cache\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:47:44.292022 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:44.291992    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-kube-api-access-szk5j" (OuterVolumeSpecName: "kube-api-access-szk5j") pod "c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58" (UID: "c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58"). InnerVolumeSpecName "kube-api-access-szk5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:47:44.292116 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:44.292052    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58" (UID: "c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:47:44.292164 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:44.292136    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-dshm" (OuterVolumeSpecName: "dshm") pod "c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58" (UID: "c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:47:44.391057 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:44.391012    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:47:44.391057 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:44.391050    2588 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-dshm\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:47:44.391057 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:44.391059    2588 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-tmp-dir\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:47:44.391057 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:44.391066    2588 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-home\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:47:44.391057 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:44.391074    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szk5j\" (UniqueName: \"kubernetes.io/projected/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58-kube-api-access-szk5j\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:47:44.462778 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:44.462745    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb" podUID="0b09b566-c9cf-4020-8dac-7399d24f9739" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:47:44.961856 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:44.961824    2588 generic.go:356] "Generic (PLEG): container finished" podID="c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58" containerID="6fb790f185094e61c5b1ae23b2329f8b2ea0cb9bc22acda7aa500c0a4631be8c" exitCode=0
Sep 17 18:47:44.962038 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:44.961869    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676" event={"ID":"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58","Type":"ContainerDied","Data":"6fb790f185094e61c5b1ae23b2329f8b2ea0cb9bc22acda7aa500c0a4631be8c"}
Sep 17 18:47:44.962038 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:44.961872    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676"
Sep 17 18:47:44.962038 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:44.961913    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676" event={"ID":"c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58","Type":"ContainerDied","Data":"b9d1e2619c974df4f423c3a092c2d498b3b2b3d3ea67eadf47c8915d97c57d27"}
Sep 17 18:47:44.962038 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:44.961937    2588 scope.go:117] "RemoveContainer" containerID="6fb790f185094e61c5b1ae23b2329f8b2ea0cb9bc22acda7aa500c0a4631be8c"
Sep 17 18:47:44.971312 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:44.971295    2588 scope.go:117] "RemoveContainer" containerID="6fb790f185094e61c5b1ae23b2329f8b2ea0cb9bc22acda7aa500c0a4631be8c"
Sep 17 18:47:44.971650 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:47:44.971625    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fb790f185094e61c5b1ae23b2329f8b2ea0cb9bc22acda7aa500c0a4631be8c\": container with ID starting with 6fb790f185094e61c5b1ae23b2329f8b2ea0cb9bc22acda7aa500c0a4631be8c not found: ID does not exist" containerID="6fb790f185094e61c5b1ae23b2329f8b2ea0cb9bc22acda7aa500c0a4631be8c"
Sep 17 18:47:44.971707 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:44.971662    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fb790f185094e61c5b1ae23b2329f8b2ea0cb9bc22acda7aa500c0a4631be8c"} err="failed to get container status \"6fb790f185094e61c5b1ae23b2329f8b2ea0cb9bc22acda7aa500c0a4631be8c\": rpc error: code = NotFound desc = could not find container \"6fb790f185094e61c5b1ae23b2329f8b2ea0cb9bc22acda7aa500c0a4631be8c\": container with ID starting with 6fb790f185094e61c5b1ae23b2329f8b2ea0cb9bc22acda7aa500c0a4631be8c not found: ID does not exist"
Sep 17 18:47:44.985995 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:44.985966    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676"]
Sep 17 18:47:44.990572 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:44.990519    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-qw676"]
Sep 17 18:47:45.506471 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:45.506441    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58" path="/var/lib/kubelet/pods/c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58/volumes"
Sep 17 18:47:54.463053 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:54.463009    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb" podUID="0b09b566-c9cf-4020-8dac-7399d24f9739" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:47:59.009392 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:47:59.009333    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb" podUID="0b09b566-c9cf-4020-8dac-7399d24f9739" containerName="main" containerID="cri-o://3c44cc98b3bee39ae34c6e79ef3d848aad186277de5de901d9bf4863199f666a" gracePeriod=45
Sep 17 18:48:04.462791 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:04.462751    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb" podUID="0b09b566-c9cf-4020-8dac-7399d24f9739" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:48:04.463201 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:04.462866    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb"
Sep 17 18:48:14.462236 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:14.462192    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb" podUID="0b09b566-c9cf-4020-8dac-7399d24f9739" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:48:24.462262 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:24.462227    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb" podUID="0b09b566-c9cf-4020-8dac-7399d24f9739" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:48:29.051065 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:48:29.051027    2588 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b09b566_c9cf_4020_8dac_7399d24f9739.slice/crio-conmon-3c44cc98b3bee39ae34c6e79ef3d848aad186277de5de901d9bf4863199f666a.scope\": RecentStats: unable to find data in memory cache]"
Sep 17 18:48:29.151844 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:29.151815    2588 generic.go:356] "Generic (PLEG): container finished" podID="0b09b566-c9cf-4020-8dac-7399d24f9739" containerID="3c44cc98b3bee39ae34c6e79ef3d848aad186277de5de901d9bf4863199f666a" exitCode=0
Sep 17 18:48:29.152005 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:29.151874    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb" event={"ID":"0b09b566-c9cf-4020-8dac-7399d24f9739","Type":"ContainerDied","Data":"3c44cc98b3bee39ae34c6e79ef3d848aad186277de5de901d9bf4863199f666a"}
Sep 17 18:48:29.254843 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:29.254818    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb"
Sep 17 18:48:29.358623 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:29.358589    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b09b566-c9cf-4020-8dac-7399d24f9739-tls-certs\") pod \"0b09b566-c9cf-4020-8dac-7399d24f9739\" (UID: \"0b09b566-c9cf-4020-8dac-7399d24f9739\") "
Sep 17 18:48:29.358806 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:29.358789    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2lct\" (UniqueName: \"kubernetes.io/projected/0b09b566-c9cf-4020-8dac-7399d24f9739-kube-api-access-x2lct\") pod \"0b09b566-c9cf-4020-8dac-7399d24f9739\" (UID: \"0b09b566-c9cf-4020-8dac-7399d24f9739\") "
Sep 17 18:48:29.360825 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:29.360800    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b09b566-c9cf-4020-8dac-7399d24f9739-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0b09b566-c9cf-4020-8dac-7399d24f9739" (UID: "0b09b566-c9cf-4020-8dac-7399d24f9739"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:48:29.360902 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:29.360804    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b09b566-c9cf-4020-8dac-7399d24f9739-kube-api-access-x2lct" (OuterVolumeSpecName: "kube-api-access-x2lct") pod "0b09b566-c9cf-4020-8dac-7399d24f9739" (UID: "0b09b566-c9cf-4020-8dac-7399d24f9739"). InnerVolumeSpecName "kube-api-access-x2lct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:48:29.460292 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:29.460258    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b09b566-c9cf-4020-8dac-7399d24f9739-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:48:29.460292 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:29.460285    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2lct\" (UniqueName: \"kubernetes.io/projected/0b09b566-c9cf-4020-8dac-7399d24f9739-kube-api-access-x2lct\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:48:30.157619 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:30.157588    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb"
Sep 17 18:48:30.158166 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:30.157617    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb" event={"ID":"0b09b566-c9cf-4020-8dac-7399d24f9739","Type":"ContainerDied","Data":"162d894fb2997d3ecd25cfd426ec168f826e739926b6e72d973fc33b5ef7b3ef"}
Sep 17 18:48:30.158166 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:30.157672    2588 scope.go:117] "RemoveContainer" containerID="3c44cc98b3bee39ae34c6e79ef3d848aad186277de5de901d9bf4863199f666a"
Sep 17 18:48:30.190914 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:30.190885    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb"]
Sep 17 18:48:30.200373 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:30.200349    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xsmgb"]
Sep 17 18:48:31.505985 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:31.505958    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b09b566-c9cf-4020-8dac-7399d24f9739" path="/var/lib/kubelet/pods/0b09b566-c9cf-4020-8dac-7399d24f9739/volumes"
Sep 17 18:48:34.094811 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:34.094775    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:34.238046 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:34.238013    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:41.180390 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:41.180355    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:43.080924 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.080892    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6"]
Sep 17 18:48:43.081287 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.081190    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58" containerName="main"
Sep 17 18:48:43.081287 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.081202    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58" containerName="main"
Sep 17 18:48:43.081287 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.081223    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b09b566-c9cf-4020-8dac-7399d24f9739" containerName="main"
Sep 17 18:48:43.081287 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.081228    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b09b566-c9cf-4020-8dac-7399d24f9739" containerName="main"
Sep 17 18:48:43.081287 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.081274    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6ddabc2-b7c8-49b0-ad82-8f64a0fd1d58" containerName="main"
Sep 17 18:48:43.081287 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.081286    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b09b566-c9cf-4020-8dac-7399d24f9739" containerName="main"
Sep 17 18:48:43.084059 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.084042    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6"
Sep 17 18:48:43.090736 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.090710    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 17 18:48:43.090861 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.090743    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 17 18:48:43.090861 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.090718    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 17 18:48:43.091136 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.091120    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-9jsw2\""
Sep 17 18:48:43.117807 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.117782    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6"]
Sep 17 18:48:43.125433 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.125406    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:43.168159 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.168133    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbhb9\" (UniqueName: \"kubernetes.io/projected/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-kube-api-access-jbhb9\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6\" (UID: \"a82bc0c7-e0ac-42c6-9f75-5b480cd90679\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6"
Sep 17 18:48:43.168310 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.168165    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6\" (UID: \"a82bc0c7-e0ac-42c6-9f75-5b480cd90679\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6"
Sep 17 18:48:43.168310 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.168184    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6\" (UID: \"a82bc0c7-e0ac-42c6-9f75-5b480cd90679\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6"
Sep 17 18:48:43.168310 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.168209    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6\" (UID: \"a82bc0c7-e0ac-42c6-9f75-5b480cd90679\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6"
Sep 17 18:48:43.168310 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.168247    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6\" (UID: \"a82bc0c7-e0ac-42c6-9f75-5b480cd90679\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6"
Sep 17 18:48:43.168446 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.168345    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6\" (UID: \"a82bc0c7-e0ac-42c6-9f75-5b480cd90679\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6"
Sep 17 18:48:43.269480 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.269450    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6\" (UID: \"a82bc0c7-e0ac-42c6-9f75-5b480cd90679\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6"
Sep 17 18:48:43.269654 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.269498    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbhb9\" (UniqueName: \"kubernetes.io/projected/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-kube-api-access-jbhb9\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6\" (UID: \"a82bc0c7-e0ac-42c6-9f75-5b480cd90679\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6"
Sep 17 18:48:43.269654 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.269546    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6\" (UID: \"a82bc0c7-e0ac-42c6-9f75-5b480cd90679\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6"
Sep 17 18:48:43.269654 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.269563    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6\" (UID: \"a82bc0c7-e0ac-42c6-9f75-5b480cd90679\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6"
Sep 17 18:48:43.269654 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.269588    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6\" (UID: \"a82bc0c7-e0ac-42c6-9f75-5b480cd90679\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6"
Sep 17 18:48:43.269654 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.269604    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6\" (UID: \"a82bc0c7-e0ac-42c6-9f75-5b480cd90679\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6"
Sep 17 18:48:43.269982 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.269920    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6\" (UID: \"a82bc0c7-e0ac-42c6-9f75-5b480cd90679\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6"
Sep 17 18:48:43.269982 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.269969    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6\" (UID: \"a82bc0c7-e0ac-42c6-9f75-5b480cd90679\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6"
Sep 17 18:48:43.270078 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.269970    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6\" (UID: \"a82bc0c7-e0ac-42c6-9f75-5b480cd90679\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6"
Sep 17 18:48:43.271995 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.271964    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6\" (UID: \"a82bc0c7-e0ac-42c6-9f75-5b480cd90679\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6"
Sep 17 18:48:43.272277 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.272261    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6\" (UID: \"a82bc0c7-e0ac-42c6-9f75-5b480cd90679\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6"
Sep 17 18:48:43.290771 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.290748    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbhb9\" (UniqueName: \"kubernetes.io/projected/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-kube-api-access-jbhb9\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6\" (UID: \"a82bc0c7-e0ac-42c6-9f75-5b480cd90679\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6"
Sep 17 18:48:43.393684 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.393596    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6"
Sep 17 18:48:43.465631 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.465599    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7w568"]
Sep 17 18:48:43.469728 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.469700    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7w568"
Sep 17 18:48:43.471004 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.470979    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5b7t\" (UniqueName: \"kubernetes.io/projected/5dd49cb4-d47e-4ec2-a98c-8e8d33ece541-kube-api-access-k5b7t\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched7w568\" (UID: \"5dd49cb4-d47e-4ec2-a98c-8e8d33ece541\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7w568"
Sep 17 18:48:43.471107 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.471056    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dd49cb4-d47e-4ec2-a98c-8e8d33ece541-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched7w568\" (UID: \"5dd49cb4-d47e-4ec2-a98c-8e8d33ece541\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7w568"
Sep 17 18:48:43.473428 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.473409    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-n7gr8\""
Sep 17 18:48:43.492633 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.492610    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7w568"]
Sep 17 18:48:43.556482 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.556449    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:43.571752 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.571718    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5b7t\" (UniqueName: \"kubernetes.io/projected/5dd49cb4-d47e-4ec2-a98c-8e8d33ece541-kube-api-access-k5b7t\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched7w568\" (UID: \"5dd49cb4-d47e-4ec2-a98c-8e8d33ece541\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7w568"
Sep 17 18:48:43.571934 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.571816    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dd49cb4-d47e-4ec2-a98c-8e8d33ece541-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched7w568\" (UID: \"5dd49cb4-d47e-4ec2-a98c-8e8d33ece541\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7w568"
Sep 17 18:48:43.574588 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.574561    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dd49cb4-d47e-4ec2-a98c-8e8d33ece541-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched7w568\" (UID: \"5dd49cb4-d47e-4ec2-a98c-8e8d33ece541\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7w568"
Sep 17 18:48:43.585188 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.585165    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5b7t\" (UniqueName: \"kubernetes.io/projected/5dd49cb4-d47e-4ec2-a98c-8e8d33ece541-kube-api-access-k5b7t\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched7w568\" (UID: \"5dd49cb4-d47e-4ec2-a98c-8e8d33ece541\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7w568"
Sep 17 18:48:43.677519 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.677498    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6"]
Sep 17 18:48:43.679138 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:48:43.679106    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda82bc0c7_e0ac_42c6_9f75_5b480cd90679.slice/crio-5cae66db529e089bdaa699ddf9fe1f01b3b91bdb9aeafb83a18ef42b6c08c417 WatchSource:0}: Error finding container 5cae66db529e089bdaa699ddf9fe1f01b3b91bdb9aeafb83a18ef42b6c08c417: Status 404 returned error can't find the container with id 5cae66db529e089bdaa699ddf9fe1f01b3b91bdb9aeafb83a18ef42b6c08c417
Sep 17 18:48:43.782186 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:43.782157    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7w568"
Sep 17 18:48:44.070790 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:44.070760    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7w568"]
Sep 17 18:48:44.074200 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:48:44.074167    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd49cb4_d47e_4ec2_a98c_8e8d33ece541.slice/crio-8a985aaaf158997c88e1e00b1ef2079d84e108a133c4160517821ac66350bf1c WatchSource:0}: Error finding container 8a985aaaf158997c88e1e00b1ef2079d84e108a133c4160517821ac66350bf1c: Status 404 returned error can't find the container with id 8a985aaaf158997c88e1e00b1ef2079d84e108a133c4160517821ac66350bf1c
Sep 17 18:48:44.220469 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:44.220427    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7w568" event={"ID":"5dd49cb4-d47e-4ec2-a98c-8e8d33ece541","Type":"ContainerStarted","Data":"00bd42cf7b605536e814720ee36e9c9493530b24e1e4b073ceb0ba59fc3c9008"}
Sep 17 18:48:44.220928 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:44.220481    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7w568" event={"ID":"5dd49cb4-d47e-4ec2-a98c-8e8d33ece541","Type":"ContainerStarted","Data":"8a985aaaf158997c88e1e00b1ef2079d84e108a133c4160517821ac66350bf1c"}
Sep 17 18:48:44.220928 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:44.220559    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7w568"
Sep 17 18:48:44.222061 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:44.222035    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6" event={"ID":"a82bc0c7-e0ac-42c6-9f75-5b480cd90679","Type":"ContainerStarted","Data":"776afa11f935a53b36bd017bd6ba18750e931b727c5059ea38b64ed82a17a37c"}
Sep 17 18:48:44.222176 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:44.222084    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6" event={"ID":"a82bc0c7-e0ac-42c6-9f75-5b480cd90679","Type":"ContainerStarted","Data":"5cae66db529e089bdaa699ddf9fe1f01b3b91bdb9aeafb83a18ef42b6c08c417"}
Sep 17 18:48:44.241499 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:44.241456    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7w568" podStartSLOduration=1.241442579 podStartE2EDuration="1.241442579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:48:44.240048742 +0000 UTC m=+1161.091054141" watchObservedRunningTime="2026-09-17 18:48:44.241442579 +0000 UTC m=+1161.092447978"
Sep 17 18:48:44.262048 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:44.262003    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6" podStartSLOduration=1.261988781 podStartE2EDuration="1.261988781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:48:44.261494905 +0000 UTC m=+1161.112500298" watchObservedRunningTime="2026-09-17 18:48:44.261988781 +0000 UTC m=+1161.112994180"
Sep 17 18:48:44.394390 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:44.394312    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6"
Sep 17 18:48:48.293260 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:48.293223    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:48.579942 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:48.579910    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:48:53.394739 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:53.394699    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6"
Sep 17 18:48:53.407097 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:53.407065    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6"
Sep 17 18:48:54.276007 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:48:54.275980    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6"
Sep 17 18:49:15.227478 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:15.227444    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7w568"
Sep 17 18:49:19.580896 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:19.580860    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:25.227508 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:25.227461    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7w568" podUID="5dd49cb4-d47e-4ec2-a98c-8e8d33ece541" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:49:25.232483 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:25.232455    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:25.237392 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:25.237364    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7w568"]
Sep 17 18:49:25.242050 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:25.242022    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6"]
Sep 17 18:49:25.250008 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:49:25.249978    2588 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:49:25.250008 ip-10-0-136-132 kubenswrapper[2588]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:49:25.250008 ip-10-0-136-132 kubenswrapper[2588]:         , stdout: , stderr: , exit code -1
Sep 17 18:49:25.250008 ip-10-0-136-132 kubenswrapper[2588]:  > containerID="776afa11f935a53b36bd017bd6ba18750e931b727c5059ea38b64ed82a17a37c" cmd=["/bin/sleep","15"]
Sep 17 18:49:25.250275 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:25.250028    2588 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6" message="" err=<
Sep 17 18:49:25.250275 ip-10-0-136-132 kubenswrapper[2588]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:49:25.250275 ip-10-0-136-132 kubenswrapper[2588]:         , stdout: , stderr: , exit code -1
Sep 17 18:49:25.250275 ip-10-0-136-132 kubenswrapper[2588]:  >
Sep 17 18:49:25.250275 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:49:25.250044    2588 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:49:25.250275 ip-10-0-136-132 kubenswrapper[2588]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:49:25.250275 ip-10-0-136-132 kubenswrapper[2588]:         , stdout: , stderr: , exit code -1
Sep 17 18:49:25.250275 ip-10-0-136-132 kubenswrapper[2588]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6" podUID="a82bc0c7-e0ac-42c6-9f75-5b480cd90679" containerName="main" containerID="cri-o://776afa11f935a53b36bd017bd6ba18750e931b727c5059ea38b64ed82a17a37c"
Sep 17 18:49:25.250275 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:25.250095    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6" podUID="a82bc0c7-e0ac-42c6-9f75-5b480cd90679" containerName="main" containerID="cri-o://776afa11f935a53b36bd017bd6ba18750e931b727c5059ea38b64ed82a17a37c" gracePeriod=60
Sep 17 18:49:25.265789 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:25.265752    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6" podUID="a82bc0c7-e0ac-42c6-9f75-5b480cd90679" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Sep 17 18:49:25.300851 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:25.300826    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:25.523763 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:25.523682    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xlklb_07ee31e0-43bb-4940-837d-969bd840492b/console-operator/1.log"
Sep 17 18:49:25.525030 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:25.525006    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xlklb_07ee31e0-43bb-4940-837d-969bd840492b/console-operator/1.log"
Sep 17 18:49:25.531184 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:25.531163    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovn-acl-logging/0.log"
Sep 17 18:49:25.531858 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:25.531841    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovn-acl-logging/0.log"
Sep 17 18:49:25.618511 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:25.618485    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6"
Sep 17 18:49:25.727407 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:25.727374    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-dshm\") pod \"a82bc0c7-e0ac-42c6-9f75-5b480cd90679\" (UID: \"a82bc0c7-e0ac-42c6-9f75-5b480cd90679\") "
Sep 17 18:49:25.727407 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:25.727410    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-home\") pod \"a82bc0c7-e0ac-42c6-9f75-5b480cd90679\" (UID: \"a82bc0c7-e0ac-42c6-9f75-5b480cd90679\") "
Sep 17 18:49:25.727693 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:25.727450    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-model-cache\") pod \"a82bc0c7-e0ac-42c6-9f75-5b480cd90679\" (UID: \"a82bc0c7-e0ac-42c6-9f75-5b480cd90679\") "
Sep 17 18:49:25.727693 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:25.727478    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbhb9\" (UniqueName: \"kubernetes.io/projected/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-kube-api-access-jbhb9\") pod \"a82bc0c7-e0ac-42c6-9f75-5b480cd90679\" (UID: \"a82bc0c7-e0ac-42c6-9f75-5b480cd90679\") "
Sep 17 18:49:25.727693 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:25.727502    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-tls-certs\") pod \"a82bc0c7-e0ac-42c6-9f75-5b480cd90679\" (UID: \"a82bc0c7-e0ac-42c6-9f75-5b480cd90679\") "
Sep 17 18:49:25.727693 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:25.727555    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-tmp-dir\") pod \"a82bc0c7-e0ac-42c6-9f75-5b480cd90679\" (UID: \"a82bc0c7-e0ac-42c6-9f75-5b480cd90679\") "
Sep 17 18:49:25.727883 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:25.727724    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-model-cache" (OuterVolumeSpecName: "model-cache") pod "a82bc0c7-e0ac-42c6-9f75-5b480cd90679" (UID: "a82bc0c7-e0ac-42c6-9f75-5b480cd90679"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:49:25.727883 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:25.727793    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-home" (OuterVolumeSpecName: "home") pod "a82bc0c7-e0ac-42c6-9f75-5b480cd90679" (UID: "a82bc0c7-e0ac-42c6-9f75-5b480cd90679"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:49:25.727883 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:25.727808    2588 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-model-cache\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:49:25.728021 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:25.727890    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a82bc0c7-e0ac-42c6-9f75-5b480cd90679" (UID: "a82bc0c7-e0ac-42c6-9f75-5b480cd90679"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:49:25.729936 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:25.729907    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-dshm" (OuterVolumeSpecName: "dshm") pod "a82bc0c7-e0ac-42c6-9f75-5b480cd90679" (UID: "a82bc0c7-e0ac-42c6-9f75-5b480cd90679"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:49:25.730068 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:25.729976    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a82bc0c7-e0ac-42c6-9f75-5b480cd90679" (UID: "a82bc0c7-e0ac-42c6-9f75-5b480cd90679"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:49:25.730068 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:25.730022    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-kube-api-access-jbhb9" (OuterVolumeSpecName: "kube-api-access-jbhb9") pod "a82bc0c7-e0ac-42c6-9f75-5b480cd90679" (UID: "a82bc0c7-e0ac-42c6-9f75-5b480cd90679"). InnerVolumeSpecName "kube-api-access-jbhb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:49:25.828228 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:25.828194    2588 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-dshm\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:49:25.828228 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:25.828222    2588 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-home\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:49:25.828228 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:25.828235    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbhb9\" (UniqueName: \"kubernetes.io/projected/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-kube-api-access-jbhb9\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:49:25.828501 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:25.828249    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:49:25.828501 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:25.828262    2588 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a82bc0c7-e0ac-42c6-9f75-5b480cd90679-tmp-dir\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:49:26.411303 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:26.411265    2588 generic.go:356] "Generic (PLEG): container finished" podID="a82bc0c7-e0ac-42c6-9f75-5b480cd90679" containerID="776afa11f935a53b36bd017bd6ba18750e931b727c5059ea38b64ed82a17a37c" exitCode=0
Sep 17 18:49:26.411760 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:26.411319    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6"
Sep 17 18:49:26.411760 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:26.411356    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6" event={"ID":"a82bc0c7-e0ac-42c6-9f75-5b480cd90679","Type":"ContainerDied","Data":"776afa11f935a53b36bd017bd6ba18750e931b727c5059ea38b64ed82a17a37c"}
Sep 17 18:49:26.411760 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:26.411413    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6" event={"ID":"a82bc0c7-e0ac-42c6-9f75-5b480cd90679","Type":"ContainerDied","Data":"5cae66db529e089bdaa699ddf9fe1f01b3b91bdb9aeafb83a18ef42b6c08c417"}
Sep 17 18:49:26.411760 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:26.411437    2588 scope.go:117] "RemoveContainer" containerID="776afa11f935a53b36bd017bd6ba18750e931b727c5059ea38b64ed82a17a37c"
Sep 17 18:49:26.420587 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:26.420570    2588 scope.go:117] "RemoveContainer" containerID="776afa11f935a53b36bd017bd6ba18750e931b727c5059ea38b64ed82a17a37c"
Sep 17 18:49:26.420866 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:49:26.420847    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"776afa11f935a53b36bd017bd6ba18750e931b727c5059ea38b64ed82a17a37c\": container with ID starting with 776afa11f935a53b36bd017bd6ba18750e931b727c5059ea38b64ed82a17a37c not found: ID does not exist" containerID="776afa11f935a53b36bd017bd6ba18750e931b727c5059ea38b64ed82a17a37c"
Sep 17 18:49:26.420933 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:26.420873    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"776afa11f935a53b36bd017bd6ba18750e931b727c5059ea38b64ed82a17a37c"} err="failed to get container status \"776afa11f935a53b36bd017bd6ba18750e931b727c5059ea38b64ed82a17a37c\": rpc error: code = NotFound desc = could not find container \"776afa11f935a53b36bd017bd6ba18750e931b727c5059ea38b64ed82a17a37c\": container with ID starting with 776afa11f935a53b36bd017bd6ba18750e931b727c5059ea38b64ed82a17a37c not found: ID does not exist"
Sep 17 18:49:26.435240 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:26.435214    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6"]
Sep 17 18:49:26.444121 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:26.441802    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444bc9z6"]
Sep 17 18:49:27.115623 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:27.115580    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:27.132018 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:27.131986    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx"]
Sep 17 18:49:27.207734 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:27.207693    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:27.505544 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:27.505455    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a82bc0c7-e0ac-42c6-9f75-5b480cd90679" path="/var/lib/kubelet/pods/a82bc0c7-e0ac-42c6-9f75-5b480cd90679/volumes"
Sep 17 18:49:29.316936 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:29.316894    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:49:34.659438 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:34.659397    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx" podUID="2757fb04-a4af-4f0b-9dbd-ff2fa4624adb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:49:35.228328 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:35.228291    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7w568" podUID="5dd49cb4-d47e-4ec2-a98c-8e8d33ece541" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:49:40.257274 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:40.257207    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7w568" podUID="5dd49cb4-d47e-4ec2-a98c-8e8d33ece541" containerName="main" containerID="cri-o://00bd42cf7b605536e814720ee36e9c9493530b24e1e4b073ceb0ba59fc3c9008" gracePeriod=45
Sep 17 18:49:42.151565 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:42.151476    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx" podUID="2757fb04-a4af-4f0b-9dbd-ff2fa4624adb" containerName="main" containerID="cri-o://e84c6cfd94ddac388a94ecee6ffcaedef323c26cfaa1cb1f39df989797c56dbc" gracePeriod=45
Sep 17 18:49:44.659292 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:44.659254    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx" podUID="2757fb04-a4af-4f0b-9dbd-ff2fa4624adb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:49:45.227669 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:45.227633    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7w568" podUID="5dd49cb4-d47e-4ec2-a98c-8e8d33ece541" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:49:45.227847 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:45.227761    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7w568"
Sep 17 18:49:54.659317 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:54.659283    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx" podUID="2757fb04-a4af-4f0b-9dbd-ff2fa4624adb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:49:54.659751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:54.659398    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx"
Sep 17 18:49:55.228331 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:49:55.228290    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7w568" podUID="5dd49cb4-d47e-4ec2-a98c-8e8d33ece541" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:50:04.659448 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:04.659407    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx" podUID="2757fb04-a4af-4f0b-9dbd-ff2fa4624adb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:50:05.228197 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:05.228159    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7w568" podUID="5dd49cb4-d47e-4ec2-a98c-8e8d33ece541" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:50:10.499355 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:10.499327    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7w568"
Sep 17 18:50:10.597799 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:10.597766    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5b7t\" (UniqueName: \"kubernetes.io/projected/5dd49cb4-d47e-4ec2-a98c-8e8d33ece541-kube-api-access-k5b7t\") pod \"5dd49cb4-d47e-4ec2-a98c-8e8d33ece541\" (UID: \"5dd49cb4-d47e-4ec2-a98c-8e8d33ece541\") "
Sep 17 18:50:10.597963 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:10.597845    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dd49cb4-d47e-4ec2-a98c-8e8d33ece541-tls-certs\") pod \"5dd49cb4-d47e-4ec2-a98c-8e8d33ece541\" (UID: \"5dd49cb4-d47e-4ec2-a98c-8e8d33ece541\") "
Sep 17 18:50:10.600226 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:10.600200    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dd49cb4-d47e-4ec2-a98c-8e8d33ece541-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5dd49cb4-d47e-4ec2-a98c-8e8d33ece541" (UID: "5dd49cb4-d47e-4ec2-a98c-8e8d33ece541"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:50:10.600336 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:10.600223    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd49cb4-d47e-4ec2-a98c-8e8d33ece541-kube-api-access-k5b7t" (OuterVolumeSpecName: "kube-api-access-k5b7t") pod "5dd49cb4-d47e-4ec2-a98c-8e8d33ece541" (UID: "5dd49cb4-d47e-4ec2-a98c-8e8d33ece541"). InnerVolumeSpecName "kube-api-access-k5b7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:50:10.602001 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:10.601979    2588 generic.go:356] "Generic (PLEG): container finished" podID="5dd49cb4-d47e-4ec2-a98c-8e8d33ece541" containerID="00bd42cf7b605536e814720ee36e9c9493530b24e1e4b073ceb0ba59fc3c9008" exitCode=0
Sep 17 18:50:10.602110 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:10.602021    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7w568"
Sep 17 18:50:10.602110 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:10.602038    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7w568" event={"ID":"5dd49cb4-d47e-4ec2-a98c-8e8d33ece541","Type":"ContainerDied","Data":"00bd42cf7b605536e814720ee36e9c9493530b24e1e4b073ceb0ba59fc3c9008"}
Sep 17 18:50:10.602110 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:10.602092    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7w568" event={"ID":"5dd49cb4-d47e-4ec2-a98c-8e8d33ece541","Type":"ContainerDied","Data":"8a985aaaf158997c88e1e00b1ef2079d84e108a133c4160517821ac66350bf1c"}
Sep 17 18:50:10.602273 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:10.602123    2588 scope.go:117] "RemoveContainer" containerID="00bd42cf7b605536e814720ee36e9c9493530b24e1e4b073ceb0ba59fc3c9008"
Sep 17 18:50:10.615286 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:10.615271    2588 scope.go:117] "RemoveContainer" containerID="00bd42cf7b605536e814720ee36e9c9493530b24e1e4b073ceb0ba59fc3c9008"
Sep 17 18:50:10.615586 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:50:10.615554    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00bd42cf7b605536e814720ee36e9c9493530b24e1e4b073ceb0ba59fc3c9008\": container with ID starting with 00bd42cf7b605536e814720ee36e9c9493530b24e1e4b073ceb0ba59fc3c9008 not found: ID does not exist" containerID="00bd42cf7b605536e814720ee36e9c9493530b24e1e4b073ceb0ba59fc3c9008"
Sep 17 18:50:10.615641 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:10.615593    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00bd42cf7b605536e814720ee36e9c9493530b24e1e4b073ceb0ba59fc3c9008"} err="failed to get container status \"00bd42cf7b605536e814720ee36e9c9493530b24e1e4b073ceb0ba59fc3c9008\": rpc error: code = NotFound desc = could not find container \"00bd42cf7b605536e814720ee36e9c9493530b24e1e4b073ceb0ba59fc3c9008\": container with ID starting with 00bd42cf7b605536e814720ee36e9c9493530b24e1e4b073ceb0ba59fc3c9008 not found: ID does not exist"
Sep 17 18:50:10.626763 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:10.626740    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7w568"]
Sep 17 18:50:10.630119 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:10.630096    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched7w568"]
Sep 17 18:50:10.699310 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:10.699288    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dd49cb4-d47e-4ec2-a98c-8e8d33ece541-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:50:10.699310 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:10.699308    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5b7t\" (UniqueName: \"kubernetes.io/projected/5dd49cb4-d47e-4ec2-a98c-8e8d33ece541-kube-api-access-k5b7t\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:50:11.505309 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:11.505280    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dd49cb4-d47e-4ec2-a98c-8e8d33ece541" path="/var/lib/kubelet/pods/5dd49cb4-d47e-4ec2-a98c-8e8d33ece541/volumes"
Sep 17 18:50:12.398851 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:12.398828    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx"
Sep 17 18:50:12.515491 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:12.515407    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2757fb04-a4af-4f0b-9dbd-ff2fa4624adb-tls-certs\") pod \"2757fb04-a4af-4f0b-9dbd-ff2fa4624adb\" (UID: \"2757fb04-a4af-4f0b-9dbd-ff2fa4624adb\") "
Sep 17 18:50:12.515491 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:12.515453    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8fd6\" (UniqueName: \"kubernetes.io/projected/2757fb04-a4af-4f0b-9dbd-ff2fa4624adb-kube-api-access-l8fd6\") pod \"2757fb04-a4af-4f0b-9dbd-ff2fa4624adb\" (UID: \"2757fb04-a4af-4f0b-9dbd-ff2fa4624adb\") "
Sep 17 18:50:12.517808 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:12.517764    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2757fb04-a4af-4f0b-9dbd-ff2fa4624adb-kube-api-access-l8fd6" (OuterVolumeSpecName: "kube-api-access-l8fd6") pod "2757fb04-a4af-4f0b-9dbd-ff2fa4624adb" (UID: "2757fb04-a4af-4f0b-9dbd-ff2fa4624adb"). InnerVolumeSpecName "kube-api-access-l8fd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:50:12.517808 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:12.517803    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2757fb04-a4af-4f0b-9dbd-ff2fa4624adb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2757fb04-a4af-4f0b-9dbd-ff2fa4624adb" (UID: "2757fb04-a4af-4f0b-9dbd-ff2fa4624adb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:50:12.612422 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:12.612389    2588 generic.go:356] "Generic (PLEG): container finished" podID="2757fb04-a4af-4f0b-9dbd-ff2fa4624adb" containerID="e84c6cfd94ddac388a94ecee6ffcaedef323c26cfaa1cb1f39df989797c56dbc" exitCode=0
Sep 17 18:50:12.612632 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:12.612430    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx"
Sep 17 18:50:12.612632 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:12.612476    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx" event={"ID":"2757fb04-a4af-4f0b-9dbd-ff2fa4624adb","Type":"ContainerDied","Data":"e84c6cfd94ddac388a94ecee6ffcaedef323c26cfaa1cb1f39df989797c56dbc"}
Sep 17 18:50:12.612632 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:12.612554    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx" event={"ID":"2757fb04-a4af-4f0b-9dbd-ff2fa4624adb","Type":"ContainerDied","Data":"71a8056c44295af9b05924956463dac5b4fe2e37b3722919982a5480614d73c6"}
Sep 17 18:50:12.612632 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:12.612588    2588 scope.go:117] "RemoveContainer" containerID="e84c6cfd94ddac388a94ecee6ffcaedef323c26cfaa1cb1f39df989797c56dbc"
Sep 17 18:50:12.616146 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:12.616124    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2757fb04-a4af-4f0b-9dbd-ff2fa4624adb-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:50:12.616146 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:12.616143    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8fd6\" (UniqueName: \"kubernetes.io/projected/2757fb04-a4af-4f0b-9dbd-ff2fa4624adb-kube-api-access-l8fd6\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:50:12.620752 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:12.620734    2588 scope.go:117] "RemoveContainer" containerID="e84c6cfd94ddac388a94ecee6ffcaedef323c26cfaa1cb1f39df989797c56dbc"
Sep 17 18:50:12.620990 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:50:12.620975    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e84c6cfd94ddac388a94ecee6ffcaedef323c26cfaa1cb1f39df989797c56dbc\": container with ID starting with e84c6cfd94ddac388a94ecee6ffcaedef323c26cfaa1cb1f39df989797c56dbc not found: ID does not exist" containerID="e84c6cfd94ddac388a94ecee6ffcaedef323c26cfaa1cb1f39df989797c56dbc"
Sep 17 18:50:12.621030 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:12.620999    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e84c6cfd94ddac388a94ecee6ffcaedef323c26cfaa1cb1f39df989797c56dbc"} err="failed to get container status \"e84c6cfd94ddac388a94ecee6ffcaedef323c26cfaa1cb1f39df989797c56dbc\": rpc error: code = NotFound desc = could not find container \"e84c6cfd94ddac388a94ecee6ffcaedef323c26cfaa1cb1f39df989797c56dbc\": container with ID starting with e84c6cfd94ddac388a94ecee6ffcaedef323c26cfaa1cb1f39df989797c56dbc not found: ID does not exist"
Sep 17 18:50:12.636303 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:12.636282    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx"]
Sep 17 18:50:12.638416 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:12.638395    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-fknxx"]
Sep 17 18:50:13.505491 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:13.505460    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2757fb04-a4af-4f0b-9dbd-ff2fa4624adb" path="/var/lib/kubelet/pods/2757fb04-a4af-4f0b-9dbd-ff2fa4624adb/volumes"
Sep 17 18:50:15.249597 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:15.249562    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:15.344412 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:15.344379    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:16.915325 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:16.915291    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:17.032304 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:17.032269    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:21.760417 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:21.760380    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:23.608334 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:23.608295    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:27.523972 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:27.523936    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:27.625686 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:27.625634    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:29.504131 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:29.504101    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:29.606703 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:29.606669    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:32.879268 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:32.879239    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"]
Sep 17 18:50:32.879681 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:32.879498    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a82bc0c7-e0ac-42c6-9f75-5b480cd90679" containerName="main"
Sep 17 18:50:32.879681 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:32.879508    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="a82bc0c7-e0ac-42c6-9f75-5b480cd90679" containerName="main"
Sep 17 18:50:32.879681 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:32.879517    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2757fb04-a4af-4f0b-9dbd-ff2fa4624adb" containerName="main"
Sep 17 18:50:32.879681 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:32.879535    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="2757fb04-a4af-4f0b-9dbd-ff2fa4624adb" containerName="main"
Sep 17 18:50:32.879681 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:32.879552    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dd49cb4-d47e-4ec2-a98c-8e8d33ece541" containerName="main"
Sep 17 18:50:32.879681 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:32.879557    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dd49cb4-d47e-4ec2-a98c-8e8d33ece541" containerName="main"
Sep 17 18:50:32.879681 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:32.879603    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dd49cb4-d47e-4ec2-a98c-8e8d33ece541" containerName="main"
Sep 17 18:50:32.879681 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:32.879610    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="2757fb04-a4af-4f0b-9dbd-ff2fa4624adb" containerName="main"
Sep 17 18:50:32.879681 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:32.879618    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="a82bc0c7-e0ac-42c6-9f75-5b480cd90679" containerName="main"
Sep 17 18:50:32.883870 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:32.883847    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"
Sep 17 18:50:32.887276 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:32.887249    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:32.888040 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:32.888022    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 17 18:50:32.888132 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:32.888046    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:50:32.888132 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:32.888073    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-4vhmb\""
Sep 17 18:50:32.888988 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:32.888974    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 17 18:50:32.889049 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:32.888993    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 17 18:50:32.889103 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:32.889058    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 17 18:50:32.900265 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:32.900241    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"]
Sep 17 18:50:32.976777 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:32.976753    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-model-cache\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx\" (UID: \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"
Sep 17 18:50:32.976883 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:32.976780    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx\" (UID: \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"
Sep 17 18:50:32.976883 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:32.976802    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx\" (UID: \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"
Sep 17 18:50:32.976883 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:32.976833    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx\" (UID: \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"
Sep 17 18:50:32.976883 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:32.976877    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-dshm\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx\" (UID: \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"
Sep 17 18:50:32.977018 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:32.976951    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx\" (UID: \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"
Sep 17 18:50:32.977018 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:32.976980    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-home\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx\" (UID: \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"
Sep 17 18:50:32.977018 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:32.977000    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfsmq\" (UniqueName: \"kubernetes.io/projected/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-kube-api-access-kfsmq\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx\" (UID: \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"
Sep 17 18:50:33.077643 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.077613    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-home\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx\" (UID: \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"
Sep 17 18:50:33.077793 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.077647    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfsmq\" (UniqueName: \"kubernetes.io/projected/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-kube-api-access-kfsmq\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx\" (UID: \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"
Sep 17 18:50:33.077793 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.077677    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-model-cache\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx\" (UID: \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"
Sep 17 18:50:33.077793 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.077693    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx\" (UID: \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"
Sep 17 18:50:33.077793 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.077716    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx\" (UID: \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"
Sep 17 18:50:33.077793 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.077747    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx\" (UID: \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"
Sep 17 18:50:33.077793 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.077771    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-dshm\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx\" (UID: \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"
Sep 17 18:50:33.078124 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.077833    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx\" (UID: \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"
Sep 17 18:50:33.078124 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.078094    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-home\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx\" (UID: \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"
Sep 17 18:50:33.078211 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.078181    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-model-cache\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx\" (UID: \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"
Sep 17 18:50:33.078275 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.078224    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx\" (UID: \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"
Sep 17 18:50:33.078275 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.078248    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx\" (UID: \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"
Sep 17 18:50:33.078436 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.078419    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx\" (UID: \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"
Sep 17 18:50:33.080117 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.080099    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-dshm\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx\" (UID: \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"
Sep 17 18:50:33.080414 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.080394    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx\" (UID: \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"
Sep 17 18:50:33.086853 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.086822    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfsmq\" (UniqueName: \"kubernetes.io/projected/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-kube-api-access-kfsmq\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx\" (UID: \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"
Sep 17 18:50:33.193733 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.193662    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"
Sep 17 18:50:33.241790 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.241749    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq"]
Sep 17 18:50:33.242740 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.242713    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:33.245356 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.245338    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq"
Sep 17 18:50:33.249207 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.248600    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-vs7gd\""
Sep 17 18:50:33.263569 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.261851    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq"]
Sep 17 18:50:33.333742 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.333706    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:50:33.379974 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.379928    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84b30886-870b-4c99-bc5a-d5daa17d1332-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq\" (UID: \"84b30886-870b-4c99-bc5a-d5daa17d1332\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq"
Sep 17 18:50:33.380167 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.379984    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c8c7\" (UniqueName: \"kubernetes.io/projected/84b30886-870b-4c99-bc5a-d5daa17d1332-kube-api-access-2c8c7\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq\" (UID: \"84b30886-870b-4c99-bc5a-d5daa17d1332\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq"
Sep 17 18:50:33.476163 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.476056    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"]
Sep 17 18:50:33.478727 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:50:33.478700    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f1d6d19_a07d_4331_9126_1c6e5c112d7e.slice/crio-c796a715d319040422aac53878c922ae74ed8598eb256901755f24ee92a4c8a8 WatchSource:0}: Error finding container c796a715d319040422aac53878c922ae74ed8598eb256901755f24ee92a4c8a8: Status 404 returned error can't find the container with id c796a715d319040422aac53878c922ae74ed8598eb256901755f24ee92a4c8a8
Sep 17 18:50:33.480642 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.480627    2588 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:50:33.480708 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.480682    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84b30886-870b-4c99-bc5a-d5daa17d1332-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq\" (UID: \"84b30886-870b-4c99-bc5a-d5daa17d1332\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq"
Sep 17 18:50:33.480753 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.480732    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c8c7\" (UniqueName: \"kubernetes.io/projected/84b30886-870b-4c99-bc5a-d5daa17d1332-kube-api-access-2c8c7\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq\" (UID: \"84b30886-870b-4c99-bc5a-d5daa17d1332\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq"
Sep 17 18:50:33.483148 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.483123    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84b30886-870b-4c99-bc5a-d5daa17d1332-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq\" (UID: \"84b30886-870b-4c99-bc5a-d5daa17d1332\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq"
Sep 17 18:50:33.489472 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.489449    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c8c7\" (UniqueName: \"kubernetes.io/projected/84b30886-870b-4c99-bc5a-d5daa17d1332-kube-api-access-2c8c7\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq\" (UID: \"84b30886-870b-4c99-bc5a-d5daa17d1332\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq"
Sep 17 18:50:33.559457 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.559433    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq"
Sep 17 18:50:33.697170 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.697139    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx" event={"ID":"4f1d6d19-a07d-4331-9126-1c6e5c112d7e","Type":"ContainerStarted","Data":"c796a715d319040422aac53878c922ae74ed8598eb256901755f24ee92a4c8a8"}
Sep 17 18:50:33.831607 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:33.831502    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq"]
Sep 17 18:50:33.836719 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:50:33.836643    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84b30886_870b_4c99_bc5a_d5daa17d1332.slice/crio-af8f7cf48709555669328a4d4e0705309751f031dfc8a601e613b2cf4b2bf35c WatchSource:0}: Error finding container af8f7cf48709555669328a4d4e0705309751f031dfc8a601e613b2cf4b2bf35c: Status 404 returned error can't find the container with id af8f7cf48709555669328a4d4e0705309751f031dfc8a601e613b2cf4b2bf35c
Sep 17 18:50:34.727893 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:34.727850    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq" event={"ID":"84b30886-870b-4c99-bc5a-d5daa17d1332","Type":"ContainerStarted","Data":"253f411cc165b246cc1dfb89137cb906f8530c4e9789f1a13f77c4b069b646a7"}
Sep 17 18:50:34.728408 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:34.727925    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq" event={"ID":"84b30886-870b-4c99-bc5a-d5daa17d1332","Type":"ContainerStarted","Data":"af8f7cf48709555669328a4d4e0705309751f031dfc8a601e613b2cf4b2bf35c"}
Sep 17 18:50:34.728408 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:34.728088    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq"
Sep 17 18:50:34.772921 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:34.772864    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq" podStartSLOduration=1.7728442960000002 podStartE2EDuration="1.772844296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:50:34.772190925 +0000 UTC m=+1271.623196323" watchObservedRunningTime="2026-09-17 18:50:34.772844296 +0000 UTC m=+1271.623849695"
Sep 17 18:50:38.749304 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:38.749265    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx" event={"ID":"4f1d6d19-a07d-4331-9126-1c6e5c112d7e","Type":"ContainerStarted","Data":"4710f2d63038db310e75b70d610af5eac3a626dc4e636e8ab9849219c02cf05b"}
Sep 17 18:50:40.759126 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:40.759090    2588 generic.go:356] "Generic (PLEG): container finished" podID="4f1d6d19-a07d-4331-9126-1c6e5c112d7e" containerID="4710f2d63038db310e75b70d610af5eac3a626dc4e636e8ab9849219c02cf05b" exitCode=0
Sep 17 18:50:40.759465 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:40.759148    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx" event={"ID":"4f1d6d19-a07d-4331-9126-1c6e5c112d7e","Type":"ContainerDied","Data":"4710f2d63038db310e75b70d610af5eac3a626dc4e636e8ab9849219c02cf05b"}
Sep 17 18:50:41.770246 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:41.770209    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx" event={"ID":"4f1d6d19-a07d-4331-9126-1c6e5c112d7e","Type":"ContainerStarted","Data":"380096e9a10507b7bc99f3173fbeabe9d1ce598516206104de7472f5d8b999a5"}
Sep 17 18:50:41.795270 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:41.795224    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx" podStartSLOduration=2.808165881 podStartE2EDuration="9.795209051s" totalImagesPullingTime="4.663467643s" totalInitContainerRuntime="2.323575527s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:50:41.79339647 +0000 UTC m=+1278.644401881" watchObservedRunningTime="2026-09-17 18:50:41.795209051 +0000 UTC m=+1278.646214453"
Sep 17 18:50:42.194470 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:42.194438    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"
Sep 17 18:50:43.194544 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:43.194500    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"
Sep 17 18:50:43.207514 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:43.207485    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"
Sep 17 18:50:43.789685 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:43.789654    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"
Sep 17 18:50:53.863905 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:50:53.863865    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:51:05.735340 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:05.735311    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq"
Sep 17 18:51:08.700443 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.700408    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"]
Sep 17 18:51:08.721162 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.721140    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"]
Sep 17 18:51:08.721314 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.721232    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"
Sep 17 18:51:08.724170 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.724142    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Sep 17 18:51:08.724170 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.724158    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:51:08.724385 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.724225    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 17 18:51:08.724385 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.724286    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 17 18:51:08.724385 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.724328    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-8trbj\""
Sep 17 18:51:08.724742 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.724722    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 17 18:51:08.728921 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.728894    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:51:08.785505 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.785478    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/929eb53b-a997-4290-aef2-3839a4170607-dshm\") pod \"auth-invalid-token-test-kserve-5dbc8bb769-8fdk4\" (UID: \"929eb53b-a997-4290-aef2-3839a4170607\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"
Sep 17 18:51:08.785636 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.785513    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/929eb53b-a997-4290-aef2-3839a4170607-tmp-dir\") pod \"auth-invalid-token-test-kserve-5dbc8bb769-8fdk4\" (UID: \"929eb53b-a997-4290-aef2-3839a4170607\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"
Sep 17 18:51:08.785636 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.785588    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/929eb53b-a997-4290-aef2-3839a4170607-home\") pod \"auth-invalid-token-test-kserve-5dbc8bb769-8fdk4\" (UID: \"929eb53b-a997-4290-aef2-3839a4170607\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"
Sep 17 18:51:08.785636 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.785605    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/929eb53b-a997-4290-aef2-3839a4170607-model-cache\") pod \"auth-invalid-token-test-kserve-5dbc8bb769-8fdk4\" (UID: \"929eb53b-a997-4290-aef2-3839a4170607\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"
Sep 17 18:51:08.785767 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.785657    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hp69\" (UniqueName: \"kubernetes.io/projected/929eb53b-a997-4290-aef2-3839a4170607-kube-api-access-7hp69\") pod \"auth-invalid-token-test-kserve-5dbc8bb769-8fdk4\" (UID: \"929eb53b-a997-4290-aef2-3839a4170607\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"
Sep 17 18:51:08.785767 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.785676    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/929eb53b-a997-4290-aef2-3839a4170607-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5dbc8bb769-8fdk4\" (UID: \"929eb53b-a997-4290-aef2-3839a4170607\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"
Sep 17 18:51:08.785767 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.785697    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/929eb53b-a997-4290-aef2-3839a4170607-cabundle-cert\") pod \"auth-invalid-token-test-kserve-5dbc8bb769-8fdk4\" (UID: \"929eb53b-a997-4290-aef2-3839a4170607\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"
Sep 17 18:51:08.785880 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.785798    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/929eb53b-a997-4290-aef2-3839a4170607-tls-certs\") pod \"auth-invalid-token-test-kserve-5dbc8bb769-8fdk4\" (UID: \"929eb53b-a997-4290-aef2-3839a4170607\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"
Sep 17 18:51:08.886151 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.886126    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/929eb53b-a997-4290-aef2-3839a4170607-home\") pod \"auth-invalid-token-test-kserve-5dbc8bb769-8fdk4\" (UID: \"929eb53b-a997-4290-aef2-3839a4170607\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"
Sep 17 18:51:08.886309 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.886156    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/929eb53b-a997-4290-aef2-3839a4170607-model-cache\") pod \"auth-invalid-token-test-kserve-5dbc8bb769-8fdk4\" (UID: \"929eb53b-a997-4290-aef2-3839a4170607\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"
Sep 17 18:51:08.886309 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.886181    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hp69\" (UniqueName: \"kubernetes.io/projected/929eb53b-a997-4290-aef2-3839a4170607-kube-api-access-7hp69\") pod \"auth-invalid-token-test-kserve-5dbc8bb769-8fdk4\" (UID: \"929eb53b-a997-4290-aef2-3839a4170607\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"
Sep 17 18:51:08.886309 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.886198    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/929eb53b-a997-4290-aef2-3839a4170607-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5dbc8bb769-8fdk4\" (UID: \"929eb53b-a997-4290-aef2-3839a4170607\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"
Sep 17 18:51:08.886309 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.886213    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/929eb53b-a997-4290-aef2-3839a4170607-cabundle-cert\") pod \"auth-invalid-token-test-kserve-5dbc8bb769-8fdk4\" (UID: \"929eb53b-a997-4290-aef2-3839a4170607\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"
Sep 17 18:51:08.886587 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.886349    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/929eb53b-a997-4290-aef2-3839a4170607-tls-certs\") pod \"auth-invalid-token-test-kserve-5dbc8bb769-8fdk4\" (UID: \"929eb53b-a997-4290-aef2-3839a4170607\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"
Sep 17 18:51:08.886587 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.886420    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/929eb53b-a997-4290-aef2-3839a4170607-dshm\") pod \"auth-invalid-token-test-kserve-5dbc8bb769-8fdk4\" (UID: \"929eb53b-a997-4290-aef2-3839a4170607\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"
Sep 17 18:51:08.886587 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.886472    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/929eb53b-a997-4290-aef2-3839a4170607-tmp-dir\") pod \"auth-invalid-token-test-kserve-5dbc8bb769-8fdk4\" (UID: \"929eb53b-a997-4290-aef2-3839a4170607\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"
Sep 17 18:51:08.886747 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.886630    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/929eb53b-a997-4290-aef2-3839a4170607-model-cache\") pod \"auth-invalid-token-test-kserve-5dbc8bb769-8fdk4\" (UID: \"929eb53b-a997-4290-aef2-3839a4170607\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"
Sep 17 18:51:08.886747 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.886700    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/929eb53b-a997-4290-aef2-3839a4170607-home\") pod \"auth-invalid-token-test-kserve-5dbc8bb769-8fdk4\" (UID: \"929eb53b-a997-4290-aef2-3839a4170607\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"
Sep 17 18:51:08.886747 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.886730    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/929eb53b-a997-4290-aef2-3839a4170607-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5dbc8bb769-8fdk4\" (UID: \"929eb53b-a997-4290-aef2-3839a4170607\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"
Sep 17 18:51:08.887005 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.886968    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/929eb53b-a997-4290-aef2-3839a4170607-tmp-dir\") pod \"auth-invalid-token-test-kserve-5dbc8bb769-8fdk4\" (UID: \"929eb53b-a997-4290-aef2-3839a4170607\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"
Sep 17 18:51:08.887468 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.887445    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/929eb53b-a997-4290-aef2-3839a4170607-cabundle-cert\") pod \"auth-invalid-token-test-kserve-5dbc8bb769-8fdk4\" (UID: \"929eb53b-a997-4290-aef2-3839a4170607\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"
Sep 17 18:51:08.888866 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.888842    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/929eb53b-a997-4290-aef2-3839a4170607-dshm\") pod \"auth-invalid-token-test-kserve-5dbc8bb769-8fdk4\" (UID: \"929eb53b-a997-4290-aef2-3839a4170607\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"
Sep 17 18:51:08.889326 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.889307    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/929eb53b-a997-4290-aef2-3839a4170607-tls-certs\") pod \"auth-invalid-token-test-kserve-5dbc8bb769-8fdk4\" (UID: \"929eb53b-a997-4290-aef2-3839a4170607\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"
Sep 17 18:51:08.899325 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.899307    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hp69\" (UniqueName: \"kubernetes.io/projected/929eb53b-a997-4290-aef2-3839a4170607-kube-api-access-7hp69\") pod \"auth-invalid-token-test-kserve-5dbc8bb769-8fdk4\" (UID: \"929eb53b-a997-4290-aef2-3839a4170607\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"
Sep 17 18:51:08.908659 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.908637    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9"]
Sep 17 18:51:08.912250 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.912233    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9"
Sep 17 18:51:08.914859 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.914835    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-625nv\""
Sep 17 18:51:08.923657 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.923638    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9"]
Sep 17 18:51:08.987618 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.987554    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp9mv\" (UniqueName: \"kubernetes.io/projected/d372e7be-90a1-42f3-815e-3bc45dde635f-kube-api-access-gp9mv\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9\" (UID: \"d372e7be-90a1-42f3-815e-3bc45dde635f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9"
Sep 17 18:51:08.987618 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:08.987596    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d372e7be-90a1-42f3-815e-3bc45dde635f-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9\" (UID: \"d372e7be-90a1-42f3-815e-3bc45dde635f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9"
Sep 17 18:51:09.032321 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:09.032298    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"
Sep 17 18:51:09.088171 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:09.088050    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp9mv\" (UniqueName: \"kubernetes.io/projected/d372e7be-90a1-42f3-815e-3bc45dde635f-kube-api-access-gp9mv\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9\" (UID: \"d372e7be-90a1-42f3-815e-3bc45dde635f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9"
Sep 17 18:51:09.088171 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:09.088105    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d372e7be-90a1-42f3-815e-3bc45dde635f-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9\" (UID: \"d372e7be-90a1-42f3-815e-3bc45dde635f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9"
Sep 17 18:51:09.091356 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:09.091328    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d372e7be-90a1-42f3-815e-3bc45dde635f-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9\" (UID: \"d372e7be-90a1-42f3-815e-3bc45dde635f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9"
Sep 17 18:51:09.096392 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:09.096373    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp9mv\" (UniqueName: \"kubernetes.io/projected/d372e7be-90a1-42f3-815e-3bc45dde635f-kube-api-access-gp9mv\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9\" (UID: \"d372e7be-90a1-42f3-815e-3bc45dde635f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9"
Sep 17 18:51:09.222054 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:09.222027    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9"
Sep 17 18:51:09.309078 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:09.309048    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"]
Sep 17 18:51:09.312298 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:51:09.312271    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod929eb53b_a997_4290_aef2_3839a4170607.slice/crio-e771fb99f7c1151bcb84595ad020e21d27f67965fd4de24885b6386325690f4a WatchSource:0}: Error finding container e771fb99f7c1151bcb84595ad020e21d27f67965fd4de24885b6386325690f4a: Status 404 returned error can't find the container with id e771fb99f7c1151bcb84595ad020e21d27f67965fd4de24885b6386325690f4a
Sep 17 18:51:09.495094 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:09.495021    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9"]
Sep 17 18:51:09.498178 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:51:09.498141    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd372e7be_90a1_42f3_815e_3bc45dde635f.slice/crio-c79ae5b452a6f2c295b05bea9f8d82a83e071afbb1edf5a9855d8ac06b013af8 WatchSource:0}: Error finding container c79ae5b452a6f2c295b05bea9f8d82a83e071afbb1edf5a9855d8ac06b013af8: Status 404 returned error can't find the container with id c79ae5b452a6f2c295b05bea9f8d82a83e071afbb1edf5a9855d8ac06b013af8
Sep 17 18:51:09.897600 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:09.897563    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4" event={"ID":"929eb53b-a997-4290-aef2-3839a4170607","Type":"ContainerStarted","Data":"eb9dec5b3b5dfbaea2a8c229d4bb37f15322b1ae27aaa0114f6aa2b677bc22c7"}
Sep 17 18:51:09.898060 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:09.897620    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4" event={"ID":"929eb53b-a997-4290-aef2-3839a4170607","Type":"ContainerStarted","Data":"e771fb99f7c1151bcb84595ad020e21d27f67965fd4de24885b6386325690f4a"}
Sep 17 18:51:09.899354 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:09.899321    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9" event={"ID":"d372e7be-90a1-42f3-815e-3bc45dde635f","Type":"ContainerStarted","Data":"f73ad3764b2ce8ce5db011936ac0a62735629c2f047c10f3cbbc5513a07601ba"}
Sep 17 18:51:09.899498 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:09.899377    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9" event={"ID":"d372e7be-90a1-42f3-815e-3bc45dde635f","Type":"ContainerStarted","Data":"c79ae5b452a6f2c295b05bea9f8d82a83e071afbb1edf5a9855d8ac06b013af8"}
Sep 17 18:51:09.899498 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:09.899418    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9"
Sep 17 18:51:09.948250 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:09.948184    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9" podStartSLOduration=1.948163055 podStartE2EDuration="1.948163055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:51:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:51:09.947003819 +0000 UTC m=+1306.798009213" watchObservedRunningTime="2026-09-17 18:51:09.948163055 +0000 UTC m=+1306.799168454"
Sep 17 18:51:10.301687 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:10.301653    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:51:11.910863 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:11.910831    2588 generic.go:356] "Generic (PLEG): container finished" podID="929eb53b-a997-4290-aef2-3839a4170607" containerID="eb9dec5b3b5dfbaea2a8c229d4bb37f15322b1ae27aaa0114f6aa2b677bc22c7" exitCode=0
Sep 17 18:51:11.911344 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:11.910906    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4" event={"ID":"929eb53b-a997-4290-aef2-3839a4170607","Type":"ContainerDied","Data":"eb9dec5b3b5dfbaea2a8c229d4bb37f15322b1ae27aaa0114f6aa2b677bc22c7"}
Sep 17 18:51:25.618035 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:25.618004    2588 scope.go:117] "RemoveContainer" containerID="370920681e4bc3657ba26f1fd20edf603576da6437bac7cd62d91f1839838891"
Sep 17 18:51:33.186965 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:33.186922    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:51:40.906559 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:40.906515    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9"
Sep 17 18:51:43.078019 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:43.077987    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4" event={"ID":"929eb53b-a997-4290-aef2-3839a4170607","Type":"ContainerStarted","Data":"3b9a9141de94b404495aec765e59a8bcc3eb958eef29cc72199609694ac2324a"}
Sep 17 18:51:43.100693 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:43.100632    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4" podStartSLOduration=2.402333542 podStartE2EDuration="35.100610996s" totalImagesPullingTime="30.539598762s" totalInitContainerRuntime="2.158678692s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:51:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:51:43.098446155 +0000 UTC m=+1339.949451556" watchObservedRunningTime="2026-09-17 18:51:43.100610996 +0000 UTC m=+1339.951616398"
Sep 17 18:51:44.032953 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:44.032905    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"
Sep 17 18:51:49.032574 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:49.032520    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"
Sep 17 18:51:49.033986 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:49.033954    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4" podUID="929eb53b-a997-4290-aef2-3839a4170607" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Sep 17 18:51:59.033612 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:51:59.033562    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4" podUID="929eb53b-a997-4290-aef2-3839a4170607" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Sep 17 18:52:09.033899 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:09.033851    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4" podUID="929eb53b-a997-4290-aef2-3839a4170607" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Sep 17 18:52:19.033500 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:19.033459    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4" podUID="929eb53b-a997-4290-aef2-3839a4170607" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Sep 17 18:52:25.686812 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:25.686779    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"]
Sep 17 18:52:25.690625 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:25.690574    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:25.692705 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:25.692681    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq"]
Sep 17 18:52:25.715480 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:52:25.715437    2588 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:52:25.715480 ip-10-0-136-132 kubenswrapper[2588]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:52:25.715480 ip-10-0-136-132 kubenswrapper[2588]:         , stdout: , stderr: , exit code -1
Sep 17 18:52:25.715480 ip-10-0-136-132 kubenswrapper[2588]:  > containerID="380096e9a10507b7bc99f3173fbeabe9d1ce598516206104de7472f5d8b999a5" cmd=["/bin/sleep","15"]
Sep 17 18:52:25.715900 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:25.715869    2588 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx" message="" err=<
Sep 17 18:52:25.715900 ip-10-0-136-132 kubenswrapper[2588]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:52:25.715900 ip-10-0-136-132 kubenswrapper[2588]:         , stdout: , stderr: , exit code -1
Sep 17 18:52:25.715900 ip-10-0-136-132 kubenswrapper[2588]:  >
Sep 17 18:52:25.716138 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:52:25.716117    2588 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:52:25.716138 ip-10-0-136-132 kubenswrapper[2588]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:52:25.716138 ip-10-0-136-132 kubenswrapper[2588]:         , stdout: , stderr: , exit code -1
Sep 17 18:52:25.716138 ip-10-0-136-132 kubenswrapper[2588]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx" podUID="4f1d6d19-a07d-4331-9126-1c6e5c112d7e" containerName="main" containerID="cri-o://380096e9a10507b7bc99f3173fbeabe9d1ce598516206104de7472f5d8b999a5"
Sep 17 18:52:25.716402 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:25.716381    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx" podUID="4f1d6d19-a07d-4331-9126-1c6e5c112d7e" containerName="main" containerID="cri-o://380096e9a10507b7bc99f3173fbeabe9d1ce598516206104de7472f5d8b999a5" gracePeriod=60
Sep 17 18:52:25.741847 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:25.741810    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq" podUID="84b30886-870b-4c99-bc5a-d5daa17d1332" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:52:25.774053 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:25.774008    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:52:25.779139 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:25.779098    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx" podUID="4f1d6d19-a07d-4331-9126-1c6e5c112d7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Sep 17 18:52:25.992489 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:25.992462    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"
Sep 17 18:52:26.081711 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.081679    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-tmp-dir\") pod \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\" (UID: \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\") "
Sep 17 18:52:26.081711 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.081714    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-tls-certs\") pod \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\" (UID: \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\") "
Sep 17 18:52:26.081982 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.081769    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-kserve-provision-location\") pod \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\" (UID: \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\") "
Sep 17 18:52:26.081982 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.081798    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfsmq\" (UniqueName: \"kubernetes.io/projected/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-kube-api-access-kfsmq\") pod \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\" (UID: \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\") "
Sep 17 18:52:26.081982 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.081833    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-model-cache\") pod \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\" (UID: \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\") "
Sep 17 18:52:26.081982 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.081851    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-cabundle-cert\") pod \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\" (UID: \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\") "
Sep 17 18:52:26.081982 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.081877    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-dshm\") pod \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\" (UID: \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\") "
Sep 17 18:52:26.081982 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.081914    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-home\") pod \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\" (UID: \"4f1d6d19-a07d-4331-9126-1c6e5c112d7e\") "
Sep 17 18:52:26.082297 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.082020    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4f1d6d19-a07d-4331-9126-1c6e5c112d7e" (UID: "4f1d6d19-a07d-4331-9126-1c6e5c112d7e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:52:26.082297 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.082185    2588 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-tmp-dir\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:52:26.082297 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.082250    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-home" (OuterVolumeSpecName: "home") pod "4f1d6d19-a07d-4331-9126-1c6e5c112d7e" (UID: "4f1d6d19-a07d-4331-9126-1c6e5c112d7e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:52:26.082447 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.082358    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-model-cache" (OuterVolumeSpecName: "model-cache") pod "4f1d6d19-a07d-4331-9126-1c6e5c112d7e" (UID: "4f1d6d19-a07d-4331-9126-1c6e5c112d7e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:52:26.082447 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.082391    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4f1d6d19-a07d-4331-9126-1c6e5c112d7e" (UID: "4f1d6d19-a07d-4331-9126-1c6e5c112d7e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:52:26.084262 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.084232    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4f1d6d19-a07d-4331-9126-1c6e5c112d7e" (UID: "4f1d6d19-a07d-4331-9126-1c6e5c112d7e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:52:26.084397 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.084261    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-dshm" (OuterVolumeSpecName: "dshm") pod "4f1d6d19-a07d-4331-9126-1c6e5c112d7e" (UID: "4f1d6d19-a07d-4331-9126-1c6e5c112d7e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:52:26.084397 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.084297    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-kube-api-access-kfsmq" (OuterVolumeSpecName: "kube-api-access-kfsmq") pod "4f1d6d19-a07d-4331-9126-1c6e5c112d7e" (UID: "4f1d6d19-a07d-4331-9126-1c6e5c112d7e"). InnerVolumeSpecName "kube-api-access-kfsmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:52:26.112563 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.112476    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4f1d6d19-a07d-4331-9126-1c6e5c112d7e" (UID: "4f1d6d19-a07d-4331-9126-1c6e5c112d7e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:52:26.182949 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.182908    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:52:26.182949 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.182952    2588 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-kserve-provision-location\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:52:26.183146 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.182969    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfsmq\" (UniqueName: \"kubernetes.io/projected/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-kube-api-access-kfsmq\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:52:26.183146 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.182985    2588 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-model-cache\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:52:26.183146 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.182999    2588 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-cabundle-cert\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:52:26.183146 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.183011    2588 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-dshm\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:52:26.183146 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.183022    2588 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f1d6d19-a07d-4331-9126-1c6e5c112d7e-home\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:52:26.275286 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.275199    2588 generic.go:356] "Generic (PLEG): container finished" podID="4f1d6d19-a07d-4331-9126-1c6e5c112d7e" containerID="380096e9a10507b7bc99f3173fbeabe9d1ce598516206104de7472f5d8b999a5" exitCode=0
Sep 17 18:52:26.275286 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.275257    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"
Sep 17 18:52:26.275481 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.275265    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx" event={"ID":"4f1d6d19-a07d-4331-9126-1c6e5c112d7e","Type":"ContainerDied","Data":"380096e9a10507b7bc99f3173fbeabe9d1ce598516206104de7472f5d8b999a5"}
Sep 17 18:52:26.275481 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.275369    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx" event={"ID":"4f1d6d19-a07d-4331-9126-1c6e5c112d7e","Type":"ContainerDied","Data":"c796a715d319040422aac53878c922ae74ed8598eb256901755f24ee92a4c8a8"}
Sep 17 18:52:26.275481 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.275392    2588 scope.go:117] "RemoveContainer" containerID="380096e9a10507b7bc99f3173fbeabe9d1ce598516206104de7472f5d8b999a5"
Sep 17 18:52:26.285133 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.285113    2588 scope.go:117] "RemoveContainer" containerID="4710f2d63038db310e75b70d610af5eac3a626dc4e636e8ab9849219c02cf05b"
Sep 17 18:52:26.293084 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.293061    2588 scope.go:117] "RemoveContainer" containerID="380096e9a10507b7bc99f3173fbeabe9d1ce598516206104de7472f5d8b999a5"
Sep 17 18:52:26.293346 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:52:26.293326    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"380096e9a10507b7bc99f3173fbeabe9d1ce598516206104de7472f5d8b999a5\": container with ID starting with 380096e9a10507b7bc99f3173fbeabe9d1ce598516206104de7472f5d8b999a5 not found: ID does not exist" containerID="380096e9a10507b7bc99f3173fbeabe9d1ce598516206104de7472f5d8b999a5"
Sep 17 18:52:26.293418 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.293352    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"380096e9a10507b7bc99f3173fbeabe9d1ce598516206104de7472f5d8b999a5"} err="failed to get container status \"380096e9a10507b7bc99f3173fbeabe9d1ce598516206104de7472f5d8b999a5\": rpc error: code = NotFound desc = could not find container \"380096e9a10507b7bc99f3173fbeabe9d1ce598516206104de7472f5d8b999a5\": container with ID starting with 380096e9a10507b7bc99f3173fbeabe9d1ce598516206104de7472f5d8b999a5 not found: ID does not exist"
Sep 17 18:52:26.293418 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.293369    2588 scope.go:117] "RemoveContainer" containerID="4710f2d63038db310e75b70d610af5eac3a626dc4e636e8ab9849219c02cf05b"
Sep 17 18:52:26.293625 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:52:26.293606    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4710f2d63038db310e75b70d610af5eac3a626dc4e636e8ab9849219c02cf05b\": container with ID starting with 4710f2d63038db310e75b70d610af5eac3a626dc4e636e8ab9849219c02cf05b not found: ID does not exist" containerID="4710f2d63038db310e75b70d610af5eac3a626dc4e636e8ab9849219c02cf05b"
Sep 17 18:52:26.293711 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.293630    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4710f2d63038db310e75b70d610af5eac3a626dc4e636e8ab9849219c02cf05b"} err="failed to get container status \"4710f2d63038db310e75b70d610af5eac3a626dc4e636e8ab9849219c02cf05b\": rpc error: code = NotFound desc = could not find container \"4710f2d63038db310e75b70d610af5eac3a626dc4e636e8ab9849219c02cf05b\": container with ID starting with 4710f2d63038db310e75b70d610af5eac3a626dc4e636e8ab9849219c02cf05b not found: ID does not exist"
Sep 17 18:52:26.297175 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.297150    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"]
Sep 17 18:52:26.300994 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:26.300969    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-b6wdx"]
Sep 17 18:52:27.511937 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:27.511903    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f1d6d19-a07d-4331-9126-1c6e5c112d7e" path="/var/lib/kubelet/pods/4f1d6d19-a07d-4331-9126-1c6e5c112d7e/volumes"
Sep 17 18:52:29.032980 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:29.032937    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4" podUID="929eb53b-a997-4290-aef2-3839a4170607" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Sep 17 18:52:35.735454 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:35.735416    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq" podUID="84b30886-870b-4c99-bc5a-d5daa17d1332" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:52:39.032833 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:39.032792    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4" podUID="929eb53b-a997-4290-aef2-3839a4170607" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Sep 17 18:52:40.731870 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:40.731802    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq" podUID="84b30886-870b-4c99-bc5a-d5daa17d1332" containerName="main" containerID="cri-o://253f411cc165b246cc1dfb89137cb906f8530c4e9789f1a13f77c4b069b646a7" gracePeriod=45
Sep 17 18:52:45.734982 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:45.734940    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq" podUID="84b30886-870b-4c99-bc5a-d5daa17d1332" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:52:45.735355 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:45.735036    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq"
Sep 17 18:52:49.033737 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:49.033698    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4" podUID="929eb53b-a997-4290-aef2-3839a4170607" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Sep 17 18:52:55.735678 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:55.735638    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq" podUID="84b30886-870b-4c99-bc5a-d5daa17d1332" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:52:59.033202 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:59.033146    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4" podUID="929eb53b-a997-4290-aef2-3839a4170607" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Sep 17 18:52:59.620770 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:52:59.620736    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:53:05.735181 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:05.735136    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq" podUID="84b30886-870b-4c99-bc5a-d5daa17d1332" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:53:09.033104 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:09.033061    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4" podUID="929eb53b-a997-4290-aef2-3839a4170607" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Sep 17 18:53:15.734780 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:53:15.734747    2588 logging.go:55] [core] [Channel #467 SubChannel #468]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.37:9003", ServerName: "10.133.0.37:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.37:9003: connect: connection refused"
Sep 17 18:53:16.735415 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:16.735359    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq" podUID="84b30886-870b-4c99-bc5a-d5daa17d1332" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.133.0.37:9003\" within 1s: context deadline exceeded"
Sep 17 18:53:19.033624 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:19.033577    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4" podUID="929eb53b-a997-4290-aef2-3839a4170607" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Sep 17 18:53:19.181719 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:19.181691    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq"
Sep 17 18:53:19.236117 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:19.236014    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2c8c7\" (UniqueName: \"kubernetes.io/projected/84b30886-870b-4c99-bc5a-d5daa17d1332-kube-api-access-2c8c7\") pod \"84b30886-870b-4c99-bc5a-d5daa17d1332\" (UID: \"84b30886-870b-4c99-bc5a-d5daa17d1332\") "
Sep 17 18:53:19.236117 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:19.236068    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84b30886-870b-4c99-bc5a-d5daa17d1332-tls-certs\") pod \"84b30886-870b-4c99-bc5a-d5daa17d1332\" (UID: \"84b30886-870b-4c99-bc5a-d5daa17d1332\") "
Sep 17 18:53:19.238328 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:19.238295    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84b30886-870b-4c99-bc5a-d5daa17d1332-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "84b30886-870b-4c99-bc5a-d5daa17d1332" (UID: "84b30886-870b-4c99-bc5a-d5daa17d1332"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:53:19.238457 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:19.238301    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b30886-870b-4c99-bc5a-d5daa17d1332-kube-api-access-2c8c7" (OuterVolumeSpecName: "kube-api-access-2c8c7") pod "84b30886-870b-4c99-bc5a-d5daa17d1332" (UID: "84b30886-870b-4c99-bc5a-d5daa17d1332"). InnerVolumeSpecName "kube-api-access-2c8c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:53:19.337356 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:19.337317    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c8c7\" (UniqueName: \"kubernetes.io/projected/84b30886-870b-4c99-bc5a-d5daa17d1332-kube-api-access-2c8c7\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:53:19.337356 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:19.337358    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84b30886-870b-4c99-bc5a-d5daa17d1332-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:53:19.535099 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:19.535066    2588 generic.go:356] "Generic (PLEG): container finished" podID="84b30886-870b-4c99-bc5a-d5daa17d1332" containerID="253f411cc165b246cc1dfb89137cb906f8530c4e9789f1a13f77c4b069b646a7" exitCode=0
Sep 17 18:53:19.535278 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:19.535116    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq"
Sep 17 18:53:19.535278 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:19.535138    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq" event={"ID":"84b30886-870b-4c99-bc5a-d5daa17d1332","Type":"ContainerDied","Data":"253f411cc165b246cc1dfb89137cb906f8530c4e9789f1a13f77c4b069b646a7"}
Sep 17 18:53:19.535278 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:19.535200    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq" event={"ID":"84b30886-870b-4c99-bc5a-d5daa17d1332","Type":"ContainerDied","Data":"af8f7cf48709555669328a4d4e0705309751f031dfc8a601e613b2cf4b2bf35c"}
Sep 17 18:53:19.535278 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:19.535230    2588 scope.go:117] "RemoveContainer" containerID="253f411cc165b246cc1dfb89137cb906f8530c4e9789f1a13f77c4b069b646a7"
Sep 17 18:53:19.543917 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:19.543895    2588 scope.go:117] "RemoveContainer" containerID="253f411cc165b246cc1dfb89137cb906f8530c4e9789f1a13f77c4b069b646a7"
Sep 17 18:53:19.544252 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:53:19.544232    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"253f411cc165b246cc1dfb89137cb906f8530c4e9789f1a13f77c4b069b646a7\": container with ID starting with 253f411cc165b246cc1dfb89137cb906f8530c4e9789f1a13f77c4b069b646a7 not found: ID does not exist" containerID="253f411cc165b246cc1dfb89137cb906f8530c4e9789f1a13f77c4b069b646a7"
Sep 17 18:53:19.544308 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:19.544260    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"253f411cc165b246cc1dfb89137cb906f8530c4e9789f1a13f77c4b069b646a7"} err="failed to get container status \"253f411cc165b246cc1dfb89137cb906f8530c4e9789f1a13f77c4b069b646a7\": rpc error: code = NotFound desc = could not find container \"253f411cc165b246cc1dfb89137cb906f8530c4e9789f1a13f77c4b069b646a7\": container with ID starting with 253f411cc165b246cc1dfb89137cb906f8530c4e9789f1a13f77c4b069b646a7 not found: ID does not exist"
Sep 17 18:53:19.550919 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:19.550888    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq"]
Sep 17 18:53:19.555211 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:19.555182    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b694p9wq"]
Sep 17 18:53:20.246612 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:20.246571    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:53:20.345910 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:20.345860    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:53:21.506494 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:21.506444    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84b30886-870b-4c99-bc5a-d5daa17d1332" path="/var/lib/kubelet/pods/84b30886-870b-4c99-bc5a-d5daa17d1332/volumes"
Sep 17 18:53:26.458081 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:26.458004    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:53:29.033144 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:29.033096    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4" podUID="929eb53b-a997-4290-aef2-3839a4170607" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Sep 17 18:53:32.044993 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:32.044956    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:53:32.110886 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:32.110848    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:53:36.860781 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:36.860741    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:53:36.861178 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:36.861037    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"]
Sep 17 18:53:36.861332 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:36.861319    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f1d6d19-a07d-4331-9126-1c6e5c112d7e" containerName="storage-initializer"
Sep 17 18:53:36.861378 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:36.861334    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f1d6d19-a07d-4331-9126-1c6e5c112d7e" containerName="storage-initializer"
Sep 17 18:53:36.861378 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:36.861344    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84b30886-870b-4c99-bc5a-d5daa17d1332" containerName="main"
Sep 17 18:53:36.861378 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:36.861351    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="84b30886-870b-4c99-bc5a-d5daa17d1332" containerName="main"
Sep 17 18:53:36.861378 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:36.861369    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f1d6d19-a07d-4331-9126-1c6e5c112d7e" containerName="main"
Sep 17 18:53:36.861378 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:36.861374    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f1d6d19-a07d-4331-9126-1c6e5c112d7e" containerName="main"
Sep 17 18:53:36.861562 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:36.861428    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f1d6d19-a07d-4331-9126-1c6e5c112d7e" containerName="main"
Sep 17 18:53:36.861562 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:36.861436    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="84b30886-870b-4c99-bc5a-d5daa17d1332" containerName="main"
Sep 17 18:53:36.867972 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:36.867948    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"
Sep 17 18:53:36.871570 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:36.871544    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 17 18:53:36.871722 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:36.871581    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:53:36.871722 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:36.871591    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 17 18:53:36.872008 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:36.871991    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-vjkxw\""
Sep 17 18:53:36.872534 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:36.872509    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 17 18:53:36.872597 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:36.872550    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 17 18:53:36.878055 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:36.878023    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"]
Sep 17 18:53:36.992962 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:36.992926    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw\" (UID: \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"
Sep 17 18:53:36.992962 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:36.992967    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww6qg\" (UniqueName: \"kubernetes.io/projected/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-kube-api-access-ww6qg\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw\" (UID: \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"
Sep 17 18:53:36.993227 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:36.993014    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-tls-certs\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw\" (UID: \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"
Sep 17 18:53:36.993227 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:36.993033    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw\" (UID: \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"
Sep 17 18:53:36.993227 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:36.993050    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw\" (UID: \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"
Sep 17 18:53:36.993227 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:36.993066    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-model-cache\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw\" (UID: \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"
Sep 17 18:53:36.993227 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:36.993202    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-home\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw\" (UID: \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"
Sep 17 18:53:36.993426 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:36.993246    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-dshm\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw\" (UID: \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"
Sep 17 18:53:37.094756 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.094716    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-tls-certs\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw\" (UID: \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"
Sep 17 18:53:37.094932 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.094769    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw\" (UID: \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"
Sep 17 18:53:37.094932 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.094798    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw\" (UID: \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"
Sep 17 18:53:37.094932 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.094825    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-model-cache\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw\" (UID: \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"
Sep 17 18:53:37.094932 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.094901    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-home\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw\" (UID: \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"
Sep 17 18:53:37.094932 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.094931    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-dshm\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw\" (UID: \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"
Sep 17 18:53:37.095212 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.095003    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw\" (UID: \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"
Sep 17 18:53:37.095212 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.095037    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww6qg\" (UniqueName: \"kubernetes.io/projected/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-kube-api-access-ww6qg\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw\" (UID: \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"
Sep 17 18:53:37.095322 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.095249    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw\" (UID: \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"
Sep 17 18:53:37.095568 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.095544    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw\" (UID: \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"
Sep 17 18:53:37.095744 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.095653    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw\" (UID: \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"
Sep 17 18:53:37.095744 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.095664    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-model-cache\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw\" (UID: \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"
Sep 17 18:53:37.095891 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.095673    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-home\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw\" (UID: \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"
Sep 17 18:53:37.097568 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.097544    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-dshm\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw\" (UID: \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"
Sep 17 18:53:37.097909 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.097891    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-tls-certs\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw\" (UID: \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"
Sep 17 18:53:37.110391 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.110357    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww6qg\" (UniqueName: \"kubernetes.io/projected/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-kube-api-access-ww6qg\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw\" (UID: \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"
Sep 17 18:53:37.156800 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.156710    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:53:37.157997 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.157967    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf"]
Sep 17 18:53:37.164061 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.164033    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf"
Sep 17 18:53:37.166571 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.166523    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-qt8vl\""
Sep 17 18:53:37.174653 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.174626    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf"]
Sep 17 18:53:37.180138 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.180110    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"
Sep 17 18:53:37.238831 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.238791    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:53:37.297055 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.297014    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37c489dd-ce34-4902-b1e4-1554081429cc-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf\" (UID: \"37c489dd-ce34-4902-b1e4-1554081429cc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf"
Sep 17 18:53:37.297245 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.297093    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv6tl\" (UniqueName: \"kubernetes.io/projected/37c489dd-ce34-4902-b1e4-1554081429cc-kube-api-access-zv6tl\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf\" (UID: \"37c489dd-ce34-4902-b1e4-1554081429cc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf"
Sep 17 18:53:37.397971 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.397928    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv6tl\" (UniqueName: \"kubernetes.io/projected/37c489dd-ce34-4902-b1e4-1554081429cc-kube-api-access-zv6tl\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf\" (UID: \"37c489dd-ce34-4902-b1e4-1554081429cc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf"
Sep 17 18:53:37.398163 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.397994    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37c489dd-ce34-4902-b1e4-1554081429cc-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf\" (UID: \"37c489dd-ce34-4902-b1e4-1554081429cc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf"
Sep 17 18:53:37.400496 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.400475    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37c489dd-ce34-4902-b1e4-1554081429cc-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf\" (UID: \"37c489dd-ce34-4902-b1e4-1554081429cc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf"
Sep 17 18:53:37.406920 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.406870    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv6tl\" (UniqueName: \"kubernetes.io/projected/37c489dd-ce34-4902-b1e4-1554081429cc-kube-api-access-zv6tl\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf\" (UID: \"37c489dd-ce34-4902-b1e4-1554081429cc\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf"
Sep 17 18:53:37.475708 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.475661    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf"
Sep 17 18:53:37.486435 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.486410    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"]
Sep 17 18:53:37.487989 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:53:37.487965    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d3cb147_72c8_4b0b_ae05_f36e94d1d860.slice/crio-a83039e6bc29f38b5e392dd92fe24fb106dc27f6a5483c0226884a1efed23479 WatchSource:0}: Error finding container a83039e6bc29f38b5e392dd92fe24fb106dc27f6a5483c0226884a1efed23479: Status 404 returned error can't find the container with id a83039e6bc29f38b5e392dd92fe24fb106dc27f6a5483c0226884a1efed23479
Sep 17 18:53:37.620622 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.620579    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw" event={"ID":"5d3cb147-72c8-4b0b-ae05-f36e94d1d860","Type":"ContainerStarted","Data":"81feab215d11da8db388ff06f9c9df05aae79e1bf2a3673d8086521f81afcca6"}
Sep 17 18:53:37.620800 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.620640    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw" event={"ID":"5d3cb147-72c8-4b0b-ae05-f36e94d1d860","Type":"ContainerStarted","Data":"a83039e6bc29f38b5e392dd92fe24fb106dc27f6a5483c0226884a1efed23479"}
Sep 17 18:53:37.757300 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:37.757270    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf"]
Sep 17 18:53:37.762687 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:53:37.762646    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37c489dd_ce34_4902_b1e4_1554081429cc.slice/crio-20f96287320936497bf97b752aa001f09219ec2d087e858e2893f13c6564e3aa WatchSource:0}: Error finding container 20f96287320936497bf97b752aa001f09219ec2d087e858e2893f13c6564e3aa: Status 404 returned error can't find the container with id 20f96287320936497bf97b752aa001f09219ec2d087e858e2893f13c6564e3aa
Sep 17 18:53:38.628076 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:38.628036    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf" event={"ID":"37c489dd-ce34-4902-b1e4-1554081429cc","Type":"ContainerStarted","Data":"301cb943c5dbae6889278ec30d67ff3464a7b8bee83daf5d09f796e16546317e"}
Sep 17 18:53:38.628442 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:38.628100    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf" event={"ID":"37c489dd-ce34-4902-b1e4-1554081429cc","Type":"ContainerStarted","Data":"20f96287320936497bf97b752aa001f09219ec2d087e858e2893f13c6564e3aa"}
Sep 17 18:53:38.628442 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:38.628220    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf"
Sep 17 18:53:38.645480 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:38.645429    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf" podStartSLOduration=1.645411328 podStartE2EDuration="1.645411328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:53:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:53:38.644271713 +0000 UTC m=+1455.495277109" watchObservedRunningTime="2026-09-17 18:53:38.645411328 +0000 UTC m=+1455.496416727"
Sep 17 18:53:39.042863 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:39.042785    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"
Sep 17 18:53:39.050574 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:39.050517    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"
Sep 17 18:53:40.640790 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:40.640755    2588 generic.go:356] "Generic (PLEG): container finished" podID="5d3cb147-72c8-4b0b-ae05-f36e94d1d860" containerID="81feab215d11da8db388ff06f9c9df05aae79e1bf2a3673d8086521f81afcca6" exitCode=0
Sep 17 18:53:40.641196 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:40.640835    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw" event={"ID":"5d3cb147-72c8-4b0b-ae05-f36e94d1d860","Type":"ContainerDied","Data":"81feab215d11da8db388ff06f9c9df05aae79e1bf2a3673d8086521f81afcca6"}
Sep 17 18:53:41.648301 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:41.648267    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw" event={"ID":"5d3cb147-72c8-4b0b-ae05-f36e94d1d860","Type":"ContainerStarted","Data":"38425a418be631aef89b5957e8e2ea17f98de51f3a3c60480af13c785993ddd2"}
Sep 17 18:53:41.669674 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:41.669617    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw" podStartSLOduration=3.338830731 podStartE2EDuration="5.669598251s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.33076752s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:53:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:53:41.667450464 +0000 UTC m=+1458.518455864" watchObservedRunningTime="2026-09-17 18:53:41.669598251 +0000 UTC m=+1458.520603650"
Sep 17 18:53:42.180312 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:42.180276    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"
Sep 17 18:53:47.180711 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:47.180665    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"
Sep 17 18:53:47.193382 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:47.193353    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"
Sep 17 18:53:47.687718 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:53:47.687684    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"
Sep 17 18:54:02.483006 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:02.482967    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:02.558406 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:02.558352    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:02.873890 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:02.873853    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:05.260431 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:05.260397    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:09.636549 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:09.636497    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf"
Sep 17 18:54:10.712139 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:10.712099    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:10.723418 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:10.723390    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"]
Sep 17 18:54:10.736705 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:10.736674    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9"]
Sep 17 18:54:10.793256 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:10.793225    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:10.906224 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:10.906190    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9" podUID="d372e7be-90a1-42f3-815e-3bc45dde635f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:54:13.957371 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:13.957321    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:14.005540 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.005486    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"]
Sep 17 18:54:14.014250 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.014217    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf"]
Sep 17 18:54:14.016885 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:54:14.016844    2588 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:54:14.016885 ip-10-0-136-132 kubenswrapper[2588]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:54:14.016885 ip-10-0-136-132 kubenswrapper[2588]:         , stdout: , stderr: , exit code -1
Sep 17 18:54:14.016885 ip-10-0-136-132 kubenswrapper[2588]:  > containerID="38425a418be631aef89b5957e8e2ea17f98de51f3a3c60480af13c785993ddd2" cmd=["/bin/sleep","15"]
Sep 17 18:54:14.017235 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.016906    2588 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw" message="" err=<
Sep 17 18:54:14.017235 ip-10-0-136-132 kubenswrapper[2588]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:54:14.017235 ip-10-0-136-132 kubenswrapper[2588]:         , stdout: , stderr: , exit code -1
Sep 17 18:54:14.017235 ip-10-0-136-132 kubenswrapper[2588]:  >
Sep 17 18:54:14.017235 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:54:14.016926    2588 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:54:14.017235 ip-10-0-136-132 kubenswrapper[2588]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:54:14.017235 ip-10-0-136-132 kubenswrapper[2588]:         , stdout: , stderr: , exit code -1
Sep 17 18:54:14.017235 ip-10-0-136-132 kubenswrapper[2588]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw" podUID="5d3cb147-72c8-4b0b-ae05-f36e94d1d860" containerName="main" containerID="cri-o://38425a418be631aef89b5957e8e2ea17f98de51f3a3c60480af13c785993ddd2"
Sep 17 18:54:14.017235 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.016970    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw" podUID="5d3cb147-72c8-4b0b-ae05-f36e94d1d860" containerName="main" containerID="cri-o://38425a418be631aef89b5957e8e2ea17f98de51f3a3c60480af13c785993ddd2" gracePeriod=60
Sep 17 18:54:14.055554 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.055502    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:54:14.386698 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.386670    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"
Sep 17 18:54:14.411009 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.410965    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-tls-certs\") pod \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\" (UID: \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\") "
Sep 17 18:54:14.411009 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.411012    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-kserve-provision-location\") pod \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\" (UID: \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\") "
Sep 17 18:54:14.411261 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.411041    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-tmp-dir\") pod \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\" (UID: \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\") "
Sep 17 18:54:14.411261 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.411064    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-cabundle-cert\") pod \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\" (UID: \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\") "
Sep 17 18:54:14.411261 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.411112    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-model-cache\") pod \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\" (UID: \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\") "
Sep 17 18:54:14.411261 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.411144    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-home\") pod \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\" (UID: \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\") "
Sep 17 18:54:14.411261 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.411206    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ww6qg\" (UniqueName: \"kubernetes.io/projected/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-kube-api-access-ww6qg\") pod \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\" (UID: \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\") "
Sep 17 18:54:14.411261 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.411237    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-dshm\") pod \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\" (UID: \"5d3cb147-72c8-4b0b-ae05-f36e94d1d860\") "
Sep 17 18:54:14.411613 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.411451    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5d3cb147-72c8-4b0b-ae05-f36e94d1d860" (UID: "5d3cb147-72c8-4b0b-ae05-f36e94d1d860"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:14.411613 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.411502    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-home" (OuterVolumeSpecName: "home") pod "5d3cb147-72c8-4b0b-ae05-f36e94d1d860" (UID: "5d3cb147-72c8-4b0b-ae05-f36e94d1d860"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:14.411613 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.411544    2588 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-tmp-dir\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:54:14.411772 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.411747    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-model-cache" (OuterVolumeSpecName: "model-cache") pod "5d3cb147-72c8-4b0b-ae05-f36e94d1d860" (UID: "5d3cb147-72c8-4b0b-ae05-f36e94d1d860"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:14.412189 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.412161    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5d3cb147-72c8-4b0b-ae05-f36e94d1d860" (UID: "5d3cb147-72c8-4b0b-ae05-f36e94d1d860"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:54:14.413931 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.413907    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-dshm" (OuterVolumeSpecName: "dshm") pod "5d3cb147-72c8-4b0b-ae05-f36e94d1d860" (UID: "5d3cb147-72c8-4b0b-ae05-f36e94d1d860"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:14.414047 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.413927    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5d3cb147-72c8-4b0b-ae05-f36e94d1d860" (UID: "5d3cb147-72c8-4b0b-ae05-f36e94d1d860"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:54:14.414397 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.414371    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-kube-api-access-ww6qg" (OuterVolumeSpecName: "kube-api-access-ww6qg") pod "5d3cb147-72c8-4b0b-ae05-f36e94d1d860" (UID: "5d3cb147-72c8-4b0b-ae05-f36e94d1d860"). InnerVolumeSpecName "kube-api-access-ww6qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:54:14.443114 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.443071    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5d3cb147-72c8-4b0b-ae05-f36e94d1d860" (UID: "5d3cb147-72c8-4b0b-ae05-f36e94d1d860"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:14.512718 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.512621    2588 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-model-cache\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:54:14.512718 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.512653    2588 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-home\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:54:14.512718 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.512663    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww6qg\" (UniqueName: \"kubernetes.io/projected/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-kube-api-access-ww6qg\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:54:14.512718 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.512673    2588 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-dshm\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:54:14.512718 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.512683    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:54:14.512718 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.512691    2588 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-kserve-provision-location\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:54:14.512718 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.512699    2588 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d3cb147-72c8-4b0b-ae05-f36e94d1d860-cabundle-cert\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:54:14.804955 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.804921    2588 generic.go:356] "Generic (PLEG): container finished" podID="5d3cb147-72c8-4b0b-ae05-f36e94d1d860" containerID="38425a418be631aef89b5957e8e2ea17f98de51f3a3c60480af13c785993ddd2" exitCode=0
Sep 17 18:54:14.805131 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.804974    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"
Sep 17 18:54:14.805131 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.804982    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw" event={"ID":"5d3cb147-72c8-4b0b-ae05-f36e94d1d860","Type":"ContainerDied","Data":"38425a418be631aef89b5957e8e2ea17f98de51f3a3c60480af13c785993ddd2"}
Sep 17 18:54:14.805131 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.805026    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw" event={"ID":"5d3cb147-72c8-4b0b-ae05-f36e94d1d860","Type":"ContainerDied","Data":"a83039e6bc29f38b5e392dd92fe24fb106dc27f6a5483c0226884a1efed23479"}
Sep 17 18:54:14.805131 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.805048    2588 scope.go:117] "RemoveContainer" containerID="38425a418be631aef89b5957e8e2ea17f98de51f3a3c60480af13c785993ddd2"
Sep 17 18:54:14.814417 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.814393    2588 scope.go:117] "RemoveContainer" containerID="81feab215d11da8db388ff06f9c9df05aae79e1bf2a3673d8086521f81afcca6"
Sep 17 18:54:14.822617 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.822583    2588 scope.go:117] "RemoveContainer" containerID="38425a418be631aef89b5957e8e2ea17f98de51f3a3c60480af13c785993ddd2"
Sep 17 18:54:14.822893 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:54:14.822873    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38425a418be631aef89b5957e8e2ea17f98de51f3a3c60480af13c785993ddd2\": container with ID starting with 38425a418be631aef89b5957e8e2ea17f98de51f3a3c60480af13c785993ddd2 not found: ID does not exist" containerID="38425a418be631aef89b5957e8e2ea17f98de51f3a3c60480af13c785993ddd2"
Sep 17 18:54:14.822961 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.822903    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38425a418be631aef89b5957e8e2ea17f98de51f3a3c60480af13c785993ddd2"} err="failed to get container status \"38425a418be631aef89b5957e8e2ea17f98de51f3a3c60480af13c785993ddd2\": rpc error: code = NotFound desc = could not find container \"38425a418be631aef89b5957e8e2ea17f98de51f3a3c60480af13c785993ddd2\": container with ID starting with 38425a418be631aef89b5957e8e2ea17f98de51f3a3c60480af13c785993ddd2 not found: ID does not exist"
Sep 17 18:54:14.822961 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.822923    2588 scope.go:117] "RemoveContainer" containerID="81feab215d11da8db388ff06f9c9df05aae79e1bf2a3673d8086521f81afcca6"
Sep 17 18:54:14.823168 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:54:14.823153    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81feab215d11da8db388ff06f9c9df05aae79e1bf2a3673d8086521f81afcca6\": container with ID starting with 81feab215d11da8db388ff06f9c9df05aae79e1bf2a3673d8086521f81afcca6 not found: ID does not exist" containerID="81feab215d11da8db388ff06f9c9df05aae79e1bf2a3673d8086521f81afcca6"
Sep 17 18:54:14.823209 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.823172    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81feab215d11da8db388ff06f9c9df05aae79e1bf2a3673d8086521f81afcca6"} err="failed to get container status \"81feab215d11da8db388ff06f9c9df05aae79e1bf2a3673d8086521f81afcca6\": rpc error: code = NotFound desc = could not find container \"81feab215d11da8db388ff06f9c9df05aae79e1bf2a3673d8086521f81afcca6\": container with ID starting with 81feab215d11da8db388ff06f9c9df05aae79e1bf2a3673d8086521f81afcca6 not found: ID does not exist"
Sep 17 18:54:14.827310 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.827285    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"]
Sep 17 18:54:14.832502 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:14.832476    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-87hbw"]
Sep 17 18:54:15.506948 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:15.506917    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d3cb147-72c8-4b0b-ae05-f36e94d1d860" path="/var/lib/kubelet/pods/5d3cb147-72c8-4b0b-ae05-f36e94d1d860/volumes"
Sep 17 18:54:19.635955 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:19.635919    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf" podUID="37c489dd-ce34-4902-b1e4-1554081429cc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:54:20.906239 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:20.906205    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9" podUID="d372e7be-90a1-42f3-815e-3bc45dde635f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:54:25.552826 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:25.552800    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xlklb_07ee31e0-43bb-4940-837d-969bd840492b/console-operator/1.log"
Sep 17 18:54:25.554239 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:25.554216    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xlklb_07ee31e0-43bb-4940-837d-969bd840492b/console-operator/1.log"
Sep 17 18:54:25.558353 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:25.558334    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovn-acl-logging/0.log"
Sep 17 18:54:25.559621 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:25.559601    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovn-acl-logging/0.log"
Sep 17 18:54:25.746256 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:25.746208    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4" podUID="929eb53b-a997-4290-aef2-3839a4170607" containerName="main" containerID="cri-o://3b9a9141de94b404495aec765e59a8bcc3eb958eef29cc72199609694ac2324a" gracePeriod=45
Sep 17 18:54:25.755375 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:25.755340    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9" podUID="d372e7be-90a1-42f3-815e-3bc45dde635f" containerName="main" containerID="cri-o://f73ad3764b2ce8ce5db011936ac0a62735629c2f047c10f3cbbc5513a07601ba" gracePeriod=45
Sep 17 18:54:26.044043 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.043940    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4" podUID="929eb53b-a997-4290-aef2-3839a4170607" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Sep 17 18:54:26.495016 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.494986    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"
Sep 17 18:54:26.616041 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.615943    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/929eb53b-a997-4290-aef2-3839a4170607-home\") pod \"929eb53b-a997-4290-aef2-3839a4170607\" (UID: \"929eb53b-a997-4290-aef2-3839a4170607\") "
Sep 17 18:54:26.616041 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.616012    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/929eb53b-a997-4290-aef2-3839a4170607-tls-certs\") pod \"929eb53b-a997-4290-aef2-3839a4170607\" (UID: \"929eb53b-a997-4290-aef2-3839a4170607\") "
Sep 17 18:54:26.616574 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.616054    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/929eb53b-a997-4290-aef2-3839a4170607-model-cache\") pod \"929eb53b-a997-4290-aef2-3839a4170607\" (UID: \"929eb53b-a997-4290-aef2-3839a4170607\") "
Sep 17 18:54:26.616574 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.616082    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/929eb53b-a997-4290-aef2-3839a4170607-tmp-dir\") pod \"929eb53b-a997-4290-aef2-3839a4170607\" (UID: \"929eb53b-a997-4290-aef2-3839a4170607\") "
Sep 17 18:54:26.616574 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.616230    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/929eb53b-a997-4290-aef2-3839a4170607-kserve-provision-location\") pod \"929eb53b-a997-4290-aef2-3839a4170607\" (UID: \"929eb53b-a997-4290-aef2-3839a4170607\") "
Sep 17 18:54:26.616574 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.616287    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/929eb53b-a997-4290-aef2-3839a4170607-cabundle-cert\") pod \"929eb53b-a997-4290-aef2-3839a4170607\" (UID: \"929eb53b-a997-4290-aef2-3839a4170607\") "
Sep 17 18:54:26.616574 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.616318    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hp69\" (UniqueName: \"kubernetes.io/projected/929eb53b-a997-4290-aef2-3839a4170607-kube-api-access-7hp69\") pod \"929eb53b-a997-4290-aef2-3839a4170607\" (UID: \"929eb53b-a997-4290-aef2-3839a4170607\") "
Sep 17 18:54:26.616574 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.616358    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/929eb53b-a997-4290-aef2-3839a4170607-dshm\") pod \"929eb53b-a997-4290-aef2-3839a4170607\" (UID: \"929eb53b-a997-4290-aef2-3839a4170607\") "
Sep 17 18:54:26.616574 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.616352    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929eb53b-a997-4290-aef2-3839a4170607-model-cache" (OuterVolumeSpecName: "model-cache") pod "929eb53b-a997-4290-aef2-3839a4170607" (UID: "929eb53b-a997-4290-aef2-3839a4170607"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:26.616948 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.616641    2588 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/929eb53b-a997-4290-aef2-3839a4170607-model-cache\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:54:26.616948 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.616725    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/929eb53b-a997-4290-aef2-3839a4170607-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "929eb53b-a997-4290-aef2-3839a4170607" (UID: "929eb53b-a997-4290-aef2-3839a4170607"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:54:26.616948 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.616764    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929eb53b-a997-4290-aef2-3839a4170607-home" (OuterVolumeSpecName: "home") pod "929eb53b-a997-4290-aef2-3839a4170607" (UID: "929eb53b-a997-4290-aef2-3839a4170607"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:26.618814 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.618769    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/929eb53b-a997-4290-aef2-3839a4170607-kube-api-access-7hp69" (OuterVolumeSpecName: "kube-api-access-7hp69") pod "929eb53b-a997-4290-aef2-3839a4170607" (UID: "929eb53b-a997-4290-aef2-3839a4170607"). InnerVolumeSpecName "kube-api-access-7hp69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:54:26.618939 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.618841    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/929eb53b-a997-4290-aef2-3839a4170607-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "929eb53b-a997-4290-aef2-3839a4170607" (UID: "929eb53b-a997-4290-aef2-3839a4170607"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:54:26.618939 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.618872    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929eb53b-a997-4290-aef2-3839a4170607-dshm" (OuterVolumeSpecName: "dshm") pod "929eb53b-a997-4290-aef2-3839a4170607" (UID: "929eb53b-a997-4290-aef2-3839a4170607"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:26.628051 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.628024    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929eb53b-a997-4290-aef2-3839a4170607-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "929eb53b-a997-4290-aef2-3839a4170607" (UID: "929eb53b-a997-4290-aef2-3839a4170607"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:26.635899 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.635870    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929eb53b-a997-4290-aef2-3839a4170607-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "929eb53b-a997-4290-aef2-3839a4170607" (UID: "929eb53b-a997-4290-aef2-3839a4170607"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:26.717849 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.717815    2588 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/929eb53b-a997-4290-aef2-3839a4170607-tmp-dir\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:54:26.717849 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.717848    2588 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/929eb53b-a997-4290-aef2-3839a4170607-kserve-provision-location\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:54:26.717849 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.717858    2588 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/929eb53b-a997-4290-aef2-3839a4170607-cabundle-cert\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:54:26.718091 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.717867    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hp69\" (UniqueName: \"kubernetes.io/projected/929eb53b-a997-4290-aef2-3839a4170607-kube-api-access-7hp69\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:54:26.718091 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.717876    2588 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/929eb53b-a997-4290-aef2-3839a4170607-dshm\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:54:26.718091 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.717884    2588 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/929eb53b-a997-4290-aef2-3839a4170607-home\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:54:26.718091 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.717891    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/929eb53b-a997-4290-aef2-3839a4170607-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:54:26.859818 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.859787    2588 generic.go:356] "Generic (PLEG): container finished" podID="929eb53b-a997-4290-aef2-3839a4170607" containerID="3b9a9141de94b404495aec765e59a8bcc3eb958eef29cc72199609694ac2324a" exitCode=0
Sep 17 18:54:26.860011 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.859845    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"
Sep 17 18:54:26.860011 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.859863    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4" event={"ID":"929eb53b-a997-4290-aef2-3839a4170607","Type":"ContainerDied","Data":"3b9a9141de94b404495aec765e59a8bcc3eb958eef29cc72199609694ac2324a"}
Sep 17 18:54:26.860011 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.859928    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4" event={"ID":"929eb53b-a997-4290-aef2-3839a4170607","Type":"ContainerDied","Data":"e771fb99f7c1151bcb84595ad020e21d27f67965fd4de24885b6386325690f4a"}
Sep 17 18:54:26.860011 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.859982    2588 scope.go:117] "RemoveContainer" containerID="3b9a9141de94b404495aec765e59a8bcc3eb958eef29cc72199609694ac2324a"
Sep 17 18:54:26.869424 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.869404    2588 scope.go:117] "RemoveContainer" containerID="eb9dec5b3b5dfbaea2a8c229d4bb37f15322b1ae27aaa0114f6aa2b677bc22c7"
Sep 17 18:54:26.877339 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.877321    2588 scope.go:117] "RemoveContainer" containerID="3b9a9141de94b404495aec765e59a8bcc3eb958eef29cc72199609694ac2324a"
Sep 17 18:54:26.877639 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:54:26.877619    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b9a9141de94b404495aec765e59a8bcc3eb958eef29cc72199609694ac2324a\": container with ID starting with 3b9a9141de94b404495aec765e59a8bcc3eb958eef29cc72199609694ac2324a not found: ID does not exist" containerID="3b9a9141de94b404495aec765e59a8bcc3eb958eef29cc72199609694ac2324a"
Sep 17 18:54:26.877693 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.877648    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b9a9141de94b404495aec765e59a8bcc3eb958eef29cc72199609694ac2324a"} err="failed to get container status \"3b9a9141de94b404495aec765e59a8bcc3eb958eef29cc72199609694ac2324a\": rpc error: code = NotFound desc = could not find container \"3b9a9141de94b404495aec765e59a8bcc3eb958eef29cc72199609694ac2324a\": container with ID starting with 3b9a9141de94b404495aec765e59a8bcc3eb958eef29cc72199609694ac2324a not found: ID does not exist"
Sep 17 18:54:26.877693 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.877665    2588 scope.go:117] "RemoveContainer" containerID="eb9dec5b3b5dfbaea2a8c229d4bb37f15322b1ae27aaa0114f6aa2b677bc22c7"
Sep 17 18:54:26.877937 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:54:26.877919    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb9dec5b3b5dfbaea2a8c229d4bb37f15322b1ae27aaa0114f6aa2b677bc22c7\": container with ID starting with eb9dec5b3b5dfbaea2a8c229d4bb37f15322b1ae27aaa0114f6aa2b677bc22c7 not found: ID does not exist" containerID="eb9dec5b3b5dfbaea2a8c229d4bb37f15322b1ae27aaa0114f6aa2b677bc22c7"
Sep 17 18:54:26.877991 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.877948    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb9dec5b3b5dfbaea2a8c229d4bb37f15322b1ae27aaa0114f6aa2b677bc22c7"} err="failed to get container status \"eb9dec5b3b5dfbaea2a8c229d4bb37f15322b1ae27aaa0114f6aa2b677bc22c7\": rpc error: code = NotFound desc = could not find container \"eb9dec5b3b5dfbaea2a8c229d4bb37f15322b1ae27aaa0114f6aa2b677bc22c7\": container with ID starting with eb9dec5b3b5dfbaea2a8c229d4bb37f15322b1ae27aaa0114f6aa2b677bc22c7 not found: ID does not exist"
Sep 17 18:54:26.883442 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.883417    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"]
Sep 17 18:54:26.889114 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:26.889092    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5dbc8bb769-8fdk4"]
Sep 17 18:54:27.505889 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:27.505857    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="929eb53b-a997-4290-aef2-3839a4170607" path="/var/lib/kubelet/pods/929eb53b-a997-4290-aef2-3839a4170607/volumes"
Sep 17 18:54:29.033652 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:29.033611    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf" podUID="37c489dd-ce34-4902-b1e4-1554081429cc" containerName="main" containerID="cri-o://301cb943c5dbae6889278ec30d67ff3464a7b8bee83daf5d09f796e16546317e" gracePeriod=45
Sep 17 18:54:29.636784 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:29.636745    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf" podUID="37c489dd-ce34-4902-b1e4-1554081429cc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:54:30.906868 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:30.906829    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9" podUID="d372e7be-90a1-42f3-815e-3bc45dde635f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:54:30.907265 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:30.906939    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9"
Sep 17 18:54:39.642016 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:39.641919    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf" podUID="37c489dd-ce34-4902-b1e4-1554081429cc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:54:39.642427 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:39.642056    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf"
Sep 17 18:54:40.906187 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:40.906147    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9" podUID="d372e7be-90a1-42f3-815e-3bc45dde635f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:54:49.635891 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:49.635855    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf" podUID="37c489dd-ce34-4902-b1e4-1554081429cc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:54:50.906728 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:50.906692    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9" podUID="d372e7be-90a1-42f3-815e-3bc45dde635f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:54:55.988100 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:55.988071    2588 generic.go:356] "Generic (PLEG): container finished" podID="d372e7be-90a1-42f3-815e-3bc45dde635f" containerID="f73ad3764b2ce8ce5db011936ac0a62735629c2f047c10f3cbbc5513a07601ba" exitCode=0
Sep 17 18:54:55.988424 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:55.988151    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9" event={"ID":"d372e7be-90a1-42f3-815e-3bc45dde635f","Type":"ContainerDied","Data":"f73ad3764b2ce8ce5db011936ac0a62735629c2f047c10f3cbbc5513a07601ba"}
Sep 17 18:54:56.004378 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:56.004358    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9"
Sep 17 18:54:56.165321 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:56.165286    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gp9mv\" (UniqueName: \"kubernetes.io/projected/d372e7be-90a1-42f3-815e-3bc45dde635f-kube-api-access-gp9mv\") pod \"d372e7be-90a1-42f3-815e-3bc45dde635f\" (UID: \"d372e7be-90a1-42f3-815e-3bc45dde635f\") "
Sep 17 18:54:56.165520 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:56.165331    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d372e7be-90a1-42f3-815e-3bc45dde635f-tls-certs\") pod \"d372e7be-90a1-42f3-815e-3bc45dde635f\" (UID: \"d372e7be-90a1-42f3-815e-3bc45dde635f\") "
Sep 17 18:54:56.167429 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:56.167399    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d372e7be-90a1-42f3-815e-3bc45dde635f-kube-api-access-gp9mv" (OuterVolumeSpecName: "kube-api-access-gp9mv") pod "d372e7be-90a1-42f3-815e-3bc45dde635f" (UID: "d372e7be-90a1-42f3-815e-3bc45dde635f"). InnerVolumeSpecName "kube-api-access-gp9mv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:54:56.167558 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:56.167423    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d372e7be-90a1-42f3-815e-3bc45dde635f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d372e7be-90a1-42f3-815e-3bc45dde635f" (UID: "d372e7be-90a1-42f3-815e-3bc45dde635f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:54:56.266236 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:56.266198    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp9mv\" (UniqueName: \"kubernetes.io/projected/d372e7be-90a1-42f3-815e-3bc45dde635f-kube-api-access-gp9mv\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:54:56.266236 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:56.266228    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d372e7be-90a1-42f3-815e-3bc45dde635f-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:54:56.994646 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:56.994613    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9" event={"ID":"d372e7be-90a1-42f3-815e-3bc45dde635f","Type":"ContainerDied","Data":"c79ae5b452a6f2c295b05bea9f8d82a83e071afbb1edf5a9855d8ac06b013af8"}
Sep 17 18:54:56.995079 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:56.994656    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9"
Sep 17 18:54:56.995079 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:56.994676    2588 scope.go:117] "RemoveContainer" containerID="f73ad3764b2ce8ce5db011936ac0a62735629c2f047c10f3cbbc5513a07601ba"
Sep 17 18:54:57.016703 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:57.016677    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9"]
Sep 17 18:54:57.020025 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:57.020003    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ck9nv9"]
Sep 17 18:54:57.505731 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:57.505693    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d372e7be-90a1-42f3-815e-3bc45dde635f" path="/var/lib/kubelet/pods/d372e7be-90a1-42f3-815e-3bc45dde635f/volumes"
Sep 17 18:54:59.278497 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:59.278475    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf"
Sep 17 18:54:59.392626 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:59.392588    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37c489dd-ce34-4902-b1e4-1554081429cc-tls-certs\") pod \"37c489dd-ce34-4902-b1e4-1554081429cc\" (UID: \"37c489dd-ce34-4902-b1e4-1554081429cc\") "
Sep 17 18:54:59.392808 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:59.392667    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zv6tl\" (UniqueName: \"kubernetes.io/projected/37c489dd-ce34-4902-b1e4-1554081429cc-kube-api-access-zv6tl\") pod \"37c489dd-ce34-4902-b1e4-1554081429cc\" (UID: \"37c489dd-ce34-4902-b1e4-1554081429cc\") "
Sep 17 18:54:59.394764 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:59.394730    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37c489dd-ce34-4902-b1e4-1554081429cc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "37c489dd-ce34-4902-b1e4-1554081429cc" (UID: "37c489dd-ce34-4902-b1e4-1554081429cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:54:59.394915 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:59.394892    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37c489dd-ce34-4902-b1e4-1554081429cc-kube-api-access-zv6tl" (OuterVolumeSpecName: "kube-api-access-zv6tl") pod "37c489dd-ce34-4902-b1e4-1554081429cc" (UID: "37c489dd-ce34-4902-b1e4-1554081429cc"). InnerVolumeSpecName "kube-api-access-zv6tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:54:59.493505 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:59.493475    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37c489dd-ce34-4902-b1e4-1554081429cc-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:54:59.493505 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:54:59.493498    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv6tl\" (UniqueName: \"kubernetes.io/projected/37c489dd-ce34-4902-b1e4-1554081429cc-kube-api-access-zv6tl\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:55:00.010258 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:00.010227    2588 generic.go:356] "Generic (PLEG): container finished" podID="37c489dd-ce34-4902-b1e4-1554081429cc" containerID="301cb943c5dbae6889278ec30d67ff3464a7b8bee83daf5d09f796e16546317e" exitCode=0
Sep 17 18:55:00.010441 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:00.010283    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf"
Sep 17 18:55:00.010441 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:00.010313    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf" event={"ID":"37c489dd-ce34-4902-b1e4-1554081429cc","Type":"ContainerDied","Data":"301cb943c5dbae6889278ec30d67ff3464a7b8bee83daf5d09f796e16546317e"}
Sep 17 18:55:00.010441 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:00.010391    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf" event={"ID":"37c489dd-ce34-4902-b1e4-1554081429cc","Type":"ContainerDied","Data":"20f96287320936497bf97b752aa001f09219ec2d087e858e2893f13c6564e3aa"}
Sep 17 18:55:00.010441 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:00.010414    2588 scope.go:117] "RemoveContainer" containerID="301cb943c5dbae6889278ec30d67ff3464a7b8bee83daf5d09f796e16546317e"
Sep 17 18:55:00.018378 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:00.018326    2588 scope.go:117] "RemoveContainer" containerID="301cb943c5dbae6889278ec30d67ff3464a7b8bee83daf5d09f796e16546317e"
Sep 17 18:55:00.018664 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:55:00.018646    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"301cb943c5dbae6889278ec30d67ff3464a7b8bee83daf5d09f796e16546317e\": container with ID starting with 301cb943c5dbae6889278ec30d67ff3464a7b8bee83daf5d09f796e16546317e not found: ID does not exist" containerID="301cb943c5dbae6889278ec30d67ff3464a7b8bee83daf5d09f796e16546317e"
Sep 17 18:55:00.018735 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:00.018673    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"301cb943c5dbae6889278ec30d67ff3464a7b8bee83daf5d09f796e16546317e"} err="failed to get container status \"301cb943c5dbae6889278ec30d67ff3464a7b8bee83daf5d09f796e16546317e\": rpc error: code = NotFound desc = could not find container \"301cb943c5dbae6889278ec30d67ff3464a7b8bee83daf5d09f796e16546317e\": container with ID starting with 301cb943c5dbae6889278ec30d67ff3464a7b8bee83daf5d09f796e16546317e not found: ID does not exist"
Sep 17 18:55:00.027046 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:00.027021    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf"]
Sep 17 18:55:00.032713 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:00.032691    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kx2jf"]
Sep 17 18:55:00.060832 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:00.060805    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:00.163784 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:00.163752    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:01.505963 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:01.505923    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37c489dd-ce34-4902-b1e4-1554081429cc" path="/var/lib/kubelet/pods/37c489dd-ce34-4902-b1e4-1554081429cc/volumes"
Sep 17 18:55:02.307913 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:02.307882    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:02.494735 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:02.494703    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87"]
Sep 17 18:55:02.495012 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:02.494997    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37c489dd-ce34-4902-b1e4-1554081429cc" containerName="main"
Sep 17 18:55:02.495058 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:02.495015    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="37c489dd-ce34-4902-b1e4-1554081429cc" containerName="main"
Sep 17 18:55:02.495058 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:02.495024    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d3cb147-72c8-4b0b-ae05-f36e94d1d860" containerName="main"
Sep 17 18:55:02.495058 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:02.495030    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d3cb147-72c8-4b0b-ae05-f36e94d1d860" containerName="main"
Sep 17 18:55:02.495058 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:02.495039    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d372e7be-90a1-42f3-815e-3bc45dde635f" containerName="main"
Sep 17 18:55:02.495058 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:02.495044    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="d372e7be-90a1-42f3-815e-3bc45dde635f" containerName="main"
Sep 17 18:55:02.495058 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:02.495053    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="929eb53b-a997-4290-aef2-3839a4170607" containerName="storage-initializer"
Sep 17 18:55:02.495058 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:02.495058    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="929eb53b-a997-4290-aef2-3839a4170607" containerName="storage-initializer"
Sep 17 18:55:02.495267 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:02.495069    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d3cb147-72c8-4b0b-ae05-f36e94d1d860" containerName="storage-initializer"
Sep 17 18:55:02.495267 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:02.495076    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d3cb147-72c8-4b0b-ae05-f36e94d1d860" containerName="storage-initializer"
Sep 17 18:55:02.495267 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:02.495089    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="929eb53b-a997-4290-aef2-3839a4170607" containerName="main"
Sep 17 18:55:02.495267 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:02.495096    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="929eb53b-a997-4290-aef2-3839a4170607" containerName="main"
Sep 17 18:55:02.495267 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:02.495145    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d3cb147-72c8-4b0b-ae05-f36e94d1d860" containerName="main"
Sep 17 18:55:02.495267 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:02.495156    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="37c489dd-ce34-4902-b1e4-1554081429cc" containerName="main"
Sep 17 18:55:02.495267 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:02.495164    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="d372e7be-90a1-42f3-815e-3bc45dde635f" containerName="main"
Sep 17 18:55:02.495267 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:02.495171    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="929eb53b-a997-4290-aef2-3839a4170607" containerName="main"
Sep 17 18:55:02.499320 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:02.499303    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87"
Sep 17 18:55:02.501827 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:02.501801    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 17 18:55:02.502779 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:02.502758    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 17 18:55:02.502889 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:02.502762    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 17 18:55:02.502889 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:02.502802    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-ctd2t\""
Sep 17 18:55:02.502889 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:02.502829    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-qgfp7\""
Sep 17 18:55:02.508146 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:02.508121    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87"]
Sep 17 18:55:02.517461 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:02.517430    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edf28a02-d42b-4892-9afd-c540d99c684a-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87\" (UID: \"edf28a02-d42b-4892-9afd-c540d99c684a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87"
Sep 17 18:55:02.517632 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:02.517473    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h4bf\" (UniqueName: \"kubernetes.io/projected/edf28a02-d42b-4892-9afd-c540d99c684a-kube-api-access-9h4bf\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87\" (UID: \"edf28a02-d42b-4892-9afd-c540d99c684a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87"
Sep 17 18:55:02.530497 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:02.530464    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:02.618730 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:02.618635    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edf28a02-d42b-4892-9afd-c540d99c684a-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87\" (UID: \"edf28a02-d42b-4892-9afd-c540d99c684a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87"
Sep 17 18:55:02.618900 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:02.618802    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h4bf\" (UniqueName: \"kubernetes.io/projected/edf28a02-d42b-4892-9afd-c540d99c684a-kube-api-access-9h4bf\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87\" (UID: \"edf28a02-d42b-4892-9afd-c540d99c684a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87"
Sep 17 18:55:02.621195 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:02.621166    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edf28a02-d42b-4892-9afd-c540d99c684a-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87\" (UID: \"edf28a02-d42b-4892-9afd-c540d99c684a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87"
Sep 17 18:55:02.627385 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:02.627362    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h4bf\" (UniqueName: \"kubernetes.io/projected/edf28a02-d42b-4892-9afd-c540d99c684a-kube-api-access-9h4bf\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87\" (UID: \"edf28a02-d42b-4892-9afd-c540d99c684a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87"
Sep 17 18:55:02.810544 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:02.810494    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87"
Sep 17 18:55:03.094540 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:03.094453    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87"]
Sep 17 18:55:03.097127 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:55:03.097096    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf28a02_d42b_4892_9afd_c540d99c684a.slice/crio-83bbe52f9ba1db6150ddb1379e14aea862262dd9f871b0a4468ac28b08204f2b WatchSource:0}: Error finding container 83bbe52f9ba1db6150ddb1379e14aea862262dd9f871b0a4468ac28b08204f2b: Status 404 returned error can't find the container with id 83bbe52f9ba1db6150ddb1379e14aea862262dd9f871b0a4468ac28b08204f2b
Sep 17 18:55:03.873512 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:03.873470    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:04.016568 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:04.016484    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:04.032104 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:04.032071    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87" event={"ID":"edf28a02-d42b-4892-9afd-c540d99c684a","Type":"ContainerStarted","Data":"1a6205c11c0a9fbd4b057c081279105011a880d098dc13faa34645856f8f9d31"}
Sep 17 18:55:04.032259 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:04.032123    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87" event={"ID":"edf28a02-d42b-4892-9afd-c540d99c684a","Type":"ContainerStarted","Data":"83bbe52f9ba1db6150ddb1379e14aea862262dd9f871b0a4468ac28b08204f2b"}
Sep 17 18:55:04.032384 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:04.032355    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87"
Sep 17 18:55:04.063968 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:04.063915    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87" podStartSLOduration=2.063894931 podStartE2EDuration="2.063894931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:55:04.06186829 +0000 UTC m=+1540.912873688" watchObservedRunningTime="2026-09-17 18:55:04.063894931 +0000 UTC m=+1540.914900329"
Sep 17 18:55:06.733223 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:06.733165    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:10.181430 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:10.181398    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:12.467776 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:12.467747    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:12.563966 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:12.563938    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:13.537498 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:13.537462    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:16.009738 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:16.009709    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:16.078143 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:16.078115    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:17.658364 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:17.658330    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:21.617953 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.617922    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"]
Sep 17 18:55:21.621082 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.621058    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"
Sep 17 18:55:21.623710 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.623677    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 17 18:55:21.623894 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.623877    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:55:21.624156 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.624133    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 17 18:55:21.624283 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.624186    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 17 18:55:21.624751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.624733    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-fl45c\""
Sep 17 18:55:21.624831 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.624764    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 17 18:55:21.637906 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.637880    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"]
Sep 17 18:55:21.652151 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.652125    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:21.664550 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.664517    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-tls-certs\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp\" (UID: \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"
Sep 17 18:55:21.664712 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.664561    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp\" (UID: \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"
Sep 17 18:55:21.664712 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.664651    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp\" (UID: \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"
Sep 17 18:55:21.664712 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.664687    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-home\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp\" (UID: \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"
Sep 17 18:55:21.664712 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.664708    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-model-cache\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp\" (UID: \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"
Sep 17 18:55:21.664877 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.664738    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-tmp-dir\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp\" (UID: \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"
Sep 17 18:55:21.664877 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.664805    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-594ld\" (UniqueName: \"kubernetes.io/projected/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-kube-api-access-594ld\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp\" (UID: \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"
Sep 17 18:55:21.664877 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.664846    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-dshm\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp\" (UID: \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"
Sep 17 18:55:21.765205 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.765175    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp\" (UID: \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"
Sep 17 18:55:21.765445 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.765209    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-home\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp\" (UID: \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"
Sep 17 18:55:21.765445 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.765234    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-model-cache\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp\" (UID: \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"
Sep 17 18:55:21.765445 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.765275    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-tmp-dir\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp\" (UID: \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"
Sep 17 18:55:21.765445 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.765314    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-594ld\" (UniqueName: \"kubernetes.io/projected/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-kube-api-access-594ld\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp\" (UID: \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"
Sep 17 18:55:21.765445 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.765342    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-dshm\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp\" (UID: \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"
Sep 17 18:55:21.765445 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.765376    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-tls-certs\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp\" (UID: \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"
Sep 17 18:55:21.765445 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.765398    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp\" (UID: \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"
Sep 17 18:55:21.765893 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.765636    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-home\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp\" (UID: \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"
Sep 17 18:55:21.765893 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.765728    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-tmp-dir\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp\" (UID: \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"
Sep 17 18:55:21.765893 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.765782    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp\" (UID: \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"
Sep 17 18:55:21.766038 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.765969    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-model-cache\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp\" (UID: \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"
Sep 17 18:55:21.766038 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.765993    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp\" (UID: \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"
Sep 17 18:55:21.767810 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.767783    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-dshm\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp\" (UID: \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"
Sep 17 18:55:21.767949 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.767927    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-tls-certs\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp\" (UID: \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"
Sep 17 18:55:21.774287 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.774255    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-594ld\" (UniqueName: \"kubernetes.io/projected/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-kube-api-access-594ld\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp\" (UID: \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"
Sep 17 18:55:21.931863 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.931778    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"
Sep 17 18:55:21.971993 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:21.971941    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:22.042019 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:22.041964    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:55:22.210135 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:22.210109    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"]
Sep 17 18:55:22.212328 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:55:22.212296    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1b5fb0e_c6ea_4d92_b1f8_11fd6dc636e3.slice/crio-8f37fd3386a788b02474b7c555961b9f0def1762089a15ac952287fc24b7dbc0 WatchSource:0}: Error finding container 8f37fd3386a788b02474b7c555961b9f0def1762089a15ac952287fc24b7dbc0: Status 404 returned error can't find the container with id 8f37fd3386a788b02474b7c555961b9f0def1762089a15ac952287fc24b7dbc0
Sep 17 18:55:23.113105 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:23.113071    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp" event={"ID":"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3","Type":"ContainerStarted","Data":"977a223c6be9db783fbcdb7a0f128d7c91bec17512afcd104b0c90e472720b2c"}
Sep 17 18:55:23.113515 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:23.113127    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp" event={"ID":"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3","Type":"ContainerStarted","Data":"8f37fd3386a788b02474b7c555961b9f0def1762089a15ac952287fc24b7dbc0"}
Sep 17 18:55:25.125969 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:25.125941    2588 generic.go:356] "Generic (PLEG): container finished" podID="e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3" containerID="977a223c6be9db783fbcdb7a0f128d7c91bec17512afcd104b0c90e472720b2c" exitCode=0
Sep 17 18:55:25.126369 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:25.125998    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp" event={"ID":"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3","Type":"ContainerDied","Data":"977a223c6be9db783fbcdb7a0f128d7c91bec17512afcd104b0c90e472720b2c"}
Sep 17 18:55:26.132262 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:26.132225    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp" event={"ID":"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3","Type":"ContainerStarted","Data":"a290bd3695e9beceee6357d92910d8a9bfd6dc3bdaeed34225c9d69ec6492715"}
Sep 17 18:55:26.154801 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:26.154752    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp" podStartSLOduration=2.995491892 podStartE2EDuration="5.154736377s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.159244485s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:55:26.153766474 +0000 UTC m=+1563.004771872" watchObservedRunningTime="2026-09-17 18:55:26.154736377 +0000 UTC m=+1563.005741775"
Sep 17 18:55:26.932627 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:26.932594    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"
Sep 17 18:55:31.932601 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:31.932570    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"
Sep 17 18:55:31.945263 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:31.945239    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"
Sep 17 18:55:32.167790 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:32.167763    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"
Sep 17 18:55:35.038766 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:35.038733    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87"
Sep 17 18:55:55.418036 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:55:55.418001    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:00.872126 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:00.872080    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:00.889122 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:00.889090    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"]
Sep 17 18:56:00.898047 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:56:00.898011    2588 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:56:00.898047 ip-10-0-136-132 kubenswrapper[2588]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:56:00.898047 ip-10-0-136-132 kubenswrapper[2588]:         , stdout: , stderr: , exit code -1
Sep 17 18:56:00.898047 ip-10-0-136-132 kubenswrapper[2588]:  > containerID="a290bd3695e9beceee6357d92910d8a9bfd6dc3bdaeed34225c9d69ec6492715" cmd=["/bin/sleep","15"]
Sep 17 18:56:00.898289 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:00.898064    2588 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp" message="" err=<
Sep 17 18:56:00.898289 ip-10-0-136-132 kubenswrapper[2588]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:56:00.898289 ip-10-0-136-132 kubenswrapper[2588]:         , stdout: , stderr: , exit code -1
Sep 17 18:56:00.898289 ip-10-0-136-132 kubenswrapper[2588]:  >
Sep 17 18:56:00.898289 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:56:00.898077    2588 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:56:00.898289 ip-10-0-136-132 kubenswrapper[2588]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:56:00.898289 ip-10-0-136-132 kubenswrapper[2588]:         , stdout: , stderr: , exit code -1
Sep 17 18:56:00.898289 ip-10-0-136-132 kubenswrapper[2588]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp" podUID="e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3" containerName="main" containerID="cri-o://a290bd3695e9beceee6357d92910d8a9bfd6dc3bdaeed34225c9d69ec6492715"
Sep 17 18:56:00.898289 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:00.898111    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp" podUID="e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3" containerName="main" containerID="cri-o://a290bd3695e9beceee6357d92910d8a9bfd6dc3bdaeed34225c9d69ec6492715" gracePeriod=60
Sep 17 18:56:00.984312 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:00.984265    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:01.157857 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.157759    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp" podUID="e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 18:56:01.243055 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.243027    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"
Sep 17 18:56:01.280599 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.280568    2588 generic.go:356] "Generic (PLEG): container finished" podID="e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3" containerID="a290bd3695e9beceee6357d92910d8a9bfd6dc3bdaeed34225c9d69ec6492715" exitCode=0
Sep 17 18:56:01.280731 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.280617    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"
Sep 17 18:56:01.280731 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.280648    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp" event={"ID":"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3","Type":"ContainerDied","Data":"a290bd3695e9beceee6357d92910d8a9bfd6dc3bdaeed34225c9d69ec6492715"}
Sep 17 18:56:01.280731 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.280700    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp" event={"ID":"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3","Type":"ContainerDied","Data":"8f37fd3386a788b02474b7c555961b9f0def1762089a15ac952287fc24b7dbc0"}
Sep 17 18:56:01.280731 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.280722    2588 scope.go:117] "RemoveContainer" containerID="a290bd3695e9beceee6357d92910d8a9bfd6dc3bdaeed34225c9d69ec6492715"
Sep 17 18:56:01.289599 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.289578    2588 scope.go:117] "RemoveContainer" containerID="977a223c6be9db783fbcdb7a0f128d7c91bec17512afcd104b0c90e472720b2c"
Sep 17 18:56:01.298465 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.298439    2588 scope.go:117] "RemoveContainer" containerID="a290bd3695e9beceee6357d92910d8a9bfd6dc3bdaeed34225c9d69ec6492715"
Sep 17 18:56:01.298882 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:56:01.298863    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a290bd3695e9beceee6357d92910d8a9bfd6dc3bdaeed34225c9d69ec6492715\": container with ID starting with a290bd3695e9beceee6357d92910d8a9bfd6dc3bdaeed34225c9d69ec6492715 not found: ID does not exist" containerID="a290bd3695e9beceee6357d92910d8a9bfd6dc3bdaeed34225c9d69ec6492715"
Sep 17 18:56:01.298968 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.298890    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a290bd3695e9beceee6357d92910d8a9bfd6dc3bdaeed34225c9d69ec6492715"} err="failed to get container status \"a290bd3695e9beceee6357d92910d8a9bfd6dc3bdaeed34225c9d69ec6492715\": rpc error: code = NotFound desc = could not find container \"a290bd3695e9beceee6357d92910d8a9bfd6dc3bdaeed34225c9d69ec6492715\": container with ID starting with a290bd3695e9beceee6357d92910d8a9bfd6dc3bdaeed34225c9d69ec6492715 not found: ID does not exist"
Sep 17 18:56:01.298968 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.298906    2588 scope.go:117] "RemoveContainer" containerID="977a223c6be9db783fbcdb7a0f128d7c91bec17512afcd104b0c90e472720b2c"
Sep 17 18:56:01.299198 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:56:01.299179    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"977a223c6be9db783fbcdb7a0f128d7c91bec17512afcd104b0c90e472720b2c\": container with ID starting with 977a223c6be9db783fbcdb7a0f128d7c91bec17512afcd104b0c90e472720b2c not found: ID does not exist" containerID="977a223c6be9db783fbcdb7a0f128d7c91bec17512afcd104b0c90e472720b2c"
Sep 17 18:56:01.299237 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.299208    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"977a223c6be9db783fbcdb7a0f128d7c91bec17512afcd104b0c90e472720b2c"} err="failed to get container status \"977a223c6be9db783fbcdb7a0f128d7c91bec17512afcd104b0c90e472720b2c\": rpc error: code = NotFound desc = could not find container \"977a223c6be9db783fbcdb7a0f128d7c91bec17512afcd104b0c90e472720b2c\": container with ID starting with 977a223c6be9db783fbcdb7a0f128d7c91bec17512afcd104b0c90e472720b2c not found: ID does not exist"
Sep 17 18:56:01.385927 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.385892    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-594ld\" (UniqueName: \"kubernetes.io/projected/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-kube-api-access-594ld\") pod \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\" (UID: \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\") "
Sep 17 18:56:01.386111 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.385934    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-dshm\") pod \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\" (UID: \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\") "
Sep 17 18:56:01.386111 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.385971    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-cabundle-cert\") pod \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\" (UID: \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\") "
Sep 17 18:56:01.386111 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.386005    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-model-cache\") pod \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\" (UID: \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\") "
Sep 17 18:56:01.386111 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.386040    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-kserve-provision-location\") pod \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\" (UID: \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\") "
Sep 17 18:56:01.386111 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.386074    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-tls-certs\") pod \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\" (UID: \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\") "
Sep 17 18:56:01.386111 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.386107    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-home\") pod \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\" (UID: \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\") "
Sep 17 18:56:01.386427 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.386149    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-tmp-dir\") pod \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\" (UID: \"e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3\") "
Sep 17 18:56:01.386427 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.386324    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-model-cache" (OuterVolumeSpecName: "model-cache") pod "e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3" (UID: "e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:01.386560 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.386467    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-home" (OuterVolumeSpecName: "home") pod "e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3" (UID: "e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:01.386628 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.386562    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3" (UID: "e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:01.386702 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.386678    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3" (UID: "e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:56:01.388387 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.388364    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-kube-api-access-594ld" (OuterVolumeSpecName: "kube-api-access-594ld") pod "e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3" (UID: "e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3"). InnerVolumeSpecName "kube-api-access-594ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:56:01.388387 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.388379    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-dshm" (OuterVolumeSpecName: "dshm") pod "e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3" (UID: "e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:01.388518 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.388385    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3" (UID: "e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:56:01.413928 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.413862    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3" (UID: "e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:01.486918 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.486889    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:56:01.486918 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.486913    2588 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-home\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:56:01.486918 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.486924    2588 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-tmp-dir\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:56:01.487134 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.486932    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-594ld\" (UniqueName: \"kubernetes.io/projected/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-kube-api-access-594ld\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:56:01.487134 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.486941    2588 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-dshm\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:56:01.487134 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.486949    2588 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-cabundle-cert\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:56:01.487134 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.486959    2588 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-model-cache\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:56:01.487134 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.486966    2588 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3-kserve-provision-location\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:56:01.610150 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.610115    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"]
Sep 17 18:56:01.612196 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:01.612170    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-2wgwp"]
Sep 17 18:56:03.506097 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:03.506064    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3" path="/var/lib/kubelet/pods/e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3/volumes"
Sep 17 18:56:11.464702 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:11.464670    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:50.324883 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:50.324847    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:50.475509 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:50.475476    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:56:56.874770 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:56:56.874740    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:57:01.081873 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:57:01.081842    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:57:01.335401 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:57:01.335320    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:57:03.500265 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:57:03.500221    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:57:03.600024 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:57:03.599990    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:57:15.528174 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:57:15.528137    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:57:22.022427 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:57:22.022395    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:57:22.032402 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:57:22.032374    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87"]
Sep 17 18:57:22.093874 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:57:22.093840    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:57:25.039204 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:57:25.039170    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87" podUID="edf28a02-d42b-4892-9afd-c540d99c684a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:57:35.038657 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:57:35.038618    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87" podUID="edf28a02-d42b-4892-9afd-c540d99c684a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:57:35.088931 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:57:35.088899    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:57:35.186727 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:57:35.186693    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:57:37.051479 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:57:37.051395    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87" podUID="edf28a02-d42b-4892-9afd-c540d99c684a" containerName="main" containerID="cri-o://1a6205c11c0a9fbd4b057c081279105011a880d098dc13faa34645856f8f9d31" gracePeriod=45
Sep 17 18:57:45.039571 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:57:45.039514    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87" podUID="edf28a02-d42b-4892-9afd-c540d99c684a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:57:45.039951 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:57:45.039664    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87"
Sep 17 18:57:55.039133 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:57:55.039093    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87" podUID="edf28a02-d42b-4892-9afd-c540d99c684a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:58:05.038497 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:05.038459    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87" podUID="edf28a02-d42b-4892-9afd-c540d99c684a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:58:07.286191 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:07.286164    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87"
Sep 17 18:58:07.350555 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:07.350456    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edf28a02-d42b-4892-9afd-c540d99c684a-tls-certs\") pod \"edf28a02-d42b-4892-9afd-c540d99c684a\" (UID: \"edf28a02-d42b-4892-9afd-c540d99c684a\") "
Sep 17 18:58:07.350555 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:07.350516    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9h4bf\" (UniqueName: \"kubernetes.io/projected/edf28a02-d42b-4892-9afd-c540d99c684a-kube-api-access-9h4bf\") pod \"edf28a02-d42b-4892-9afd-c540d99c684a\" (UID: \"edf28a02-d42b-4892-9afd-c540d99c684a\") "
Sep 17 18:58:07.352669 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:07.352643    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edf28a02-d42b-4892-9afd-c540d99c684a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "edf28a02-d42b-4892-9afd-c540d99c684a" (UID: "edf28a02-d42b-4892-9afd-c540d99c684a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:58:07.352751 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:07.352643    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edf28a02-d42b-4892-9afd-c540d99c684a-kube-api-access-9h4bf" (OuterVolumeSpecName: "kube-api-access-9h4bf") pod "edf28a02-d42b-4892-9afd-c540d99c684a" (UID: "edf28a02-d42b-4892-9afd-c540d99c684a"). InnerVolumeSpecName "kube-api-access-9h4bf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:58:07.451979 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:07.451951    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edf28a02-d42b-4892-9afd-c540d99c684a-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:58:07.451979 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:07.451976    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h4bf\" (UniqueName: \"kubernetes.io/projected/edf28a02-d42b-4892-9afd-c540d99c684a-kube-api-access-9h4bf\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 18:58:07.775272 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:07.775182    2588 generic.go:356] "Generic (PLEG): container finished" podID="edf28a02-d42b-4892-9afd-c540d99c684a" containerID="1a6205c11c0a9fbd4b057c081279105011a880d098dc13faa34645856f8f9d31" exitCode=0
Sep 17 18:58:07.775272 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:07.775244    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87"
Sep 17 18:58:07.775451 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:07.775267    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87" event={"ID":"edf28a02-d42b-4892-9afd-c540d99c684a","Type":"ContainerDied","Data":"1a6205c11c0a9fbd4b057c081279105011a880d098dc13faa34645856f8f9d31"}
Sep 17 18:58:07.775451 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:07.775346    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87" event={"ID":"edf28a02-d42b-4892-9afd-c540d99c684a","Type":"ContainerDied","Data":"83bbe52f9ba1db6150ddb1379e14aea862262dd9f871b0a4468ac28b08204f2b"}
Sep 17 18:58:07.775451 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:07.775370    2588 scope.go:117] "RemoveContainer" containerID="1a6205c11c0a9fbd4b057c081279105011a880d098dc13faa34645856f8f9d31"
Sep 17 18:58:07.783569 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:07.783549    2588 scope.go:117] "RemoveContainer" containerID="1a6205c11c0a9fbd4b057c081279105011a880d098dc13faa34645856f8f9d31"
Sep 17 18:58:07.783827 ip-10-0-136-132 kubenswrapper[2588]: E0917 18:58:07.783807    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a6205c11c0a9fbd4b057c081279105011a880d098dc13faa34645856f8f9d31\": container with ID starting with 1a6205c11c0a9fbd4b057c081279105011a880d098dc13faa34645856f8f9d31 not found: ID does not exist" containerID="1a6205c11c0a9fbd4b057c081279105011a880d098dc13faa34645856f8f9d31"
Sep 17 18:58:07.783894 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:07.783839    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a6205c11c0a9fbd4b057c081279105011a880d098dc13faa34645856f8f9d31"} err="failed to get container status \"1a6205c11c0a9fbd4b057c081279105011a880d098dc13faa34645856f8f9d31\": rpc error: code = NotFound desc = could not find container \"1a6205c11c0a9fbd4b057c081279105011a880d098dc13faa34645856f8f9d31\": container with ID starting with 1a6205c11c0a9fbd4b057c081279105011a880d098dc13faa34645856f8f9d31 not found: ID does not exist"
Sep 17 18:58:07.793002 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:07.792977    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87"]
Sep 17 18:58:07.797468 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:07.797445    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-m8m87"]
Sep 17 18:58:09.506536 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:09.506506    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edf28a02-d42b-4892-9afd-c540d99c684a" path="/var/lib/kubelet/pods/edf28a02-d42b-4892-9afd-c540d99c684a/volumes"
Sep 17 18:58:11.999291 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:11.999262    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:12.128494 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.128463    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:12.360031 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.359999    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"]
Sep 17 18:58:12.360422 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.360404    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3" containerName="storage-initializer"
Sep 17 18:58:12.360485 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.360427    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3" containerName="storage-initializer"
Sep 17 18:58:12.360485 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.360457    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3" containerName="main"
Sep 17 18:58:12.360485 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.360466    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3" containerName="main"
Sep 17 18:58:12.360485 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.360482    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edf28a02-d42b-4892-9afd-c540d99c684a" containerName="main"
Sep 17 18:58:12.360663 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.360491    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="edf28a02-d42b-4892-9afd-c540d99c684a" containerName="main"
Sep 17 18:58:12.360663 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.360586    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="edf28a02-d42b-4892-9afd-c540d99c684a" containerName="main"
Sep 17 18:58:12.360663 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.360602    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1b5fb0e-c6ea-4d92-b1f8-11fd6dc636e3" containerName="main"
Sep 17 18:58:12.364989 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.364972    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:12.369663 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.369642    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Sep 17 18:58:12.369780 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.369673    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 17 18:58:12.369780 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.369699    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-q9txm\""
Sep 17 18:58:12.370061 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.370046    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 17 18:58:12.388252 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.388231    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"]
Sep 17 18:58:12.419034 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.419013    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:12.491814 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.491784    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c92f25ed-3214-4ce8-9d43-cadf9064d08e-istio-envoy\") pod \"router-gateway-1-openshift-default-8cd48b867-6rvm7\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:12.491976 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.491816    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c92f25ed-3214-4ce8-9d43-cadf9064d08e-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-8cd48b867-6rvm7\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:12.491976 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.491842    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92f25ed-3214-4ce8-9d43-cadf9064d08e-credential-socket\") pod \"router-gateway-1-openshift-default-8cd48b867-6rvm7\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:12.491976 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.491862    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92f25ed-3214-4ce8-9d43-cadf9064d08e-workload-socket\") pod \"router-gateway-1-openshift-default-8cd48b867-6rvm7\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:12.491976 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.491920    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c92f25ed-3214-4ce8-9d43-cadf9064d08e-workload-certs\") pod \"router-gateway-1-openshift-default-8cd48b867-6rvm7\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:12.492204 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.492026    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgqfj\" (UniqueName: \"kubernetes.io/projected/c92f25ed-3214-4ce8-9d43-cadf9064d08e-kube-api-access-sgqfj\") pod \"router-gateway-1-openshift-default-8cd48b867-6rvm7\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:12.492204 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.492064    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c92f25ed-3214-4ce8-9d43-cadf9064d08e-istio-podinfo\") pod \"router-gateway-1-openshift-default-8cd48b867-6rvm7\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:12.492204 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.492100    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c92f25ed-3214-4ce8-9d43-cadf9064d08e-istio-token\") pod \"router-gateway-1-openshift-default-8cd48b867-6rvm7\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:12.492204 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.492127    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c92f25ed-3214-4ce8-9d43-cadf9064d08e-istio-data\") pod \"router-gateway-1-openshift-default-8cd48b867-6rvm7\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:12.509338 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.509313    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:12.592491 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.592454    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c92f25ed-3214-4ce8-9d43-cadf9064d08e-istio-envoy\") pod \"router-gateway-1-openshift-default-8cd48b867-6rvm7\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:12.592673 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.592496    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c92f25ed-3214-4ce8-9d43-cadf9064d08e-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-8cd48b867-6rvm7\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:12.592673 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.592518    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92f25ed-3214-4ce8-9d43-cadf9064d08e-credential-socket\") pod \"router-gateway-1-openshift-default-8cd48b867-6rvm7\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:12.592673 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.592572    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92f25ed-3214-4ce8-9d43-cadf9064d08e-workload-socket\") pod \"router-gateway-1-openshift-default-8cd48b867-6rvm7\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:12.592822 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.592681    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c92f25ed-3214-4ce8-9d43-cadf9064d08e-workload-certs\") pod \"router-gateway-1-openshift-default-8cd48b867-6rvm7\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:12.592885 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.592851    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgqfj\" (UniqueName: \"kubernetes.io/projected/c92f25ed-3214-4ce8-9d43-cadf9064d08e-kube-api-access-sgqfj\") pod \"router-gateway-1-openshift-default-8cd48b867-6rvm7\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:12.592940 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.592899    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c92f25ed-3214-4ce8-9d43-cadf9064d08e-istio-podinfo\") pod \"router-gateway-1-openshift-default-8cd48b867-6rvm7\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:12.592994 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.592938    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c92f25ed-3214-4ce8-9d43-cadf9064d08e-istio-token\") pod \"router-gateway-1-openshift-default-8cd48b867-6rvm7\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:12.592994 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.592944    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92f25ed-3214-4ce8-9d43-cadf9064d08e-workload-socket\") pod \"router-gateway-1-openshift-default-8cd48b867-6rvm7\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:12.592994 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.592966    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c92f25ed-3214-4ce8-9d43-cadf9064d08e-istio-data\") pod \"router-gateway-1-openshift-default-8cd48b867-6rvm7\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:12.592994 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.592980    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92f25ed-3214-4ce8-9d43-cadf9064d08e-credential-socket\") pod \"router-gateway-1-openshift-default-8cd48b867-6rvm7\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:12.593272 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.593247    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c92f25ed-3214-4ce8-9d43-cadf9064d08e-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-8cd48b867-6rvm7\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:12.593384 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.593253    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c92f25ed-3214-4ce8-9d43-cadf9064d08e-workload-certs\") pod \"router-gateway-1-openshift-default-8cd48b867-6rvm7\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:12.593384 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.593281    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c92f25ed-3214-4ce8-9d43-cadf9064d08e-istio-data\") pod \"router-gateway-1-openshift-default-8cd48b867-6rvm7\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:12.594942 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.594922    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c92f25ed-3214-4ce8-9d43-cadf9064d08e-istio-envoy\") pod \"router-gateway-1-openshift-default-8cd48b867-6rvm7\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:12.595245 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.595228    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c92f25ed-3214-4ce8-9d43-cadf9064d08e-istio-podinfo\") pod \"router-gateway-1-openshift-default-8cd48b867-6rvm7\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:12.605019 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.604963    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c92f25ed-3214-4ce8-9d43-cadf9064d08e-istio-token\") pod \"router-gateway-1-openshift-default-8cd48b867-6rvm7\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:12.605123 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.605105    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgqfj\" (UniqueName: \"kubernetes.io/projected/c92f25ed-3214-4ce8-9d43-cadf9064d08e-kube-api-access-sgqfj\") pod \"router-gateway-1-openshift-default-8cd48b867-6rvm7\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:12.674731 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.674688    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:12.964478 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:58:12.964430    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc92f25ed_3214_4ce8_9d43_cadf9064d08e.slice/crio-060a742e21c62a9101851766cef2874af8a95300cd6f191519fff1041adb320c WatchSource:0}: Error finding container 060a742e21c62a9101851766cef2874af8a95300cd6f191519fff1041adb320c: Status 404 returned error can't find the container with id 060a742e21c62a9101851766cef2874af8a95300cd6f191519fff1041adb320c
Sep 17 18:58:12.966407 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.966321    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"]
Sep 17 18:58:12.966825 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.966797    2588 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 18:58:12.966895 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.966858    2588 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 18:58:12.966942 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:12.966908    2588 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 18:58:13.803019 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:13.802980    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7" event={"ID":"c92f25ed-3214-4ce8-9d43-cadf9064d08e","Type":"ContainerStarted","Data":"7616181c0f623af018fc474529e95b9ad87712158fcf1cc3b31ffe04aa2429d5"}
Sep 17 18:58:13.803391 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:13.803034    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7" event={"ID":"c92f25ed-3214-4ce8-9d43-cadf9064d08e","Type":"ContainerStarted","Data":"060a742e21c62a9101851766cef2874af8a95300cd6f191519fff1041adb320c"}
Sep 17 18:58:13.830399 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:13.830351    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7" podStartSLOduration=1.8303366589999999 podStartE2EDuration="1.830336659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:58:13.828498827 +0000 UTC m=+1730.679504229" watchObservedRunningTime="2026-09-17 18:58:13.830336659 +0000 UTC m=+1730.681342058"
Sep 17 18:58:14.675843 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:14.675800    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:14.680584 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:14.680560    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:14.806849 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:14.806816    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:14.807812 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:14.807791    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:15.855742 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:15.855714    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:15.952726 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:15.952695    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:18.657910 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:18.657880    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:24.561519 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:24.561481    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:24.640011 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:24.639982    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:25.115267 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:25.115239    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:25.224812 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:25.224779    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:25.860595 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:25.860564    2588 generic.go:356] "Generic (PLEG): container finished" podID="c92f25ed-3214-4ce8-9d43-cadf9064d08e" containerID="7616181c0f623af018fc474529e95b9ad87712158fcf1cc3b31ffe04aa2429d5" exitCode=0
Sep 17 18:58:25.861024 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:25.860615    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7" event={"ID":"c92f25ed-3214-4ce8-9d43-cadf9064d08e","Type":"ContainerDied","Data":"7616181c0f623af018fc474529e95b9ad87712158fcf1cc3b31ffe04aa2429d5"}
Sep 17 18:58:25.861024 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:25.860952    2588 scope.go:117] "RemoveContainer" containerID="7616181c0f623af018fc474529e95b9ad87712158fcf1cc3b31ffe04aa2429d5"
Sep 17 18:58:25.861906 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:25.861877    2588 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 18:58:25.862000 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:25.861946    2588 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 18:58:25.862000 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:25.861981    2588 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 18:58:26.866762 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:26.866717    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7" event={"ID":"c92f25ed-3214-4ce8-9d43-cadf9064d08e","Type":"ContainerStarted","Data":"64ce808a47bf3d99aff38529954241010f5d8d8177d0a36291d3743cff18851b"}
Sep 17 18:58:27.675434 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:27.675397    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:27.676382 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:27.676350    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7" podUID="c92f25ed-3214-4ce8-9d43-cadf9064d08e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 17 18:58:28.675247 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:28.675201    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7" podUID="c92f25ed-3214-4ce8-9d43-cadf9064d08e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 17 18:58:29.675547 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:29.675489    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7" podUID="c92f25ed-3214-4ce8-9d43-cadf9064d08e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 17 18:58:29.807196 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:29.807160    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:30.676066 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:30.676026    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7" podUID="c92f25ed-3214-4ce8-9d43-cadf9064d08e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 17 18:58:31.626706 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:31.626673    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:31.675219 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:31.675190    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7" podUID="c92f25ed-3214-4ce8-9d43-cadf9064d08e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 17 18:58:32.678540 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:32.678499    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:32.679514 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:32.679496    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 18:58:37.498615 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:37.498582    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:37.613493 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:37.613462    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:50.482075 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.482037    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t"]
Sep 17 18:58:50.485161 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.485139    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t"
Sep 17 18:58:50.490704 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.490669    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-wlg4n\""
Sep 17 18:58:50.490704 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.490694    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 17 18:58:50.490904 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.490768    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 17 18:58:50.490904 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.490817    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 17 18:58:50.502812 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.502787    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f90e64a-713f-470e-80bc-aad88bdebf84-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t\" (UID: \"3f90e64a-713f-470e-80bc-aad88bdebf84\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t"
Sep 17 18:58:50.502920 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.502823    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f90e64a-713f-470e-80bc-aad88bdebf84-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t\" (UID: \"3f90e64a-713f-470e-80bc-aad88bdebf84\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t"
Sep 17 18:58:50.502920 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.502847    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4xr2\" (UniqueName: \"kubernetes.io/projected/3f90e64a-713f-470e-80bc-aad88bdebf84-kube-api-access-p4xr2\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t\" (UID: \"3f90e64a-713f-470e-80bc-aad88bdebf84\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t"
Sep 17 18:58:50.503008 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.502915    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f90e64a-713f-470e-80bc-aad88bdebf84-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t\" (UID: \"3f90e64a-713f-470e-80bc-aad88bdebf84\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t"
Sep 17 18:58:50.503008 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.502979    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f90e64a-713f-470e-80bc-aad88bdebf84-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t\" (UID: \"3f90e64a-713f-470e-80bc-aad88bdebf84\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t"
Sep 17 18:58:50.503087 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.503006    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f90e64a-713f-470e-80bc-aad88bdebf84-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t\" (UID: \"3f90e64a-713f-470e-80bc-aad88bdebf84\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t"
Sep 17 18:58:50.560187 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.560158    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t"]
Sep 17 18:58:50.573124 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.573094    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:50.603385 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.603353    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f90e64a-713f-470e-80bc-aad88bdebf84-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t\" (UID: \"3f90e64a-713f-470e-80bc-aad88bdebf84\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t"
Sep 17 18:58:50.603555 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.603412    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f90e64a-713f-470e-80bc-aad88bdebf84-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t\" (UID: \"3f90e64a-713f-470e-80bc-aad88bdebf84\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t"
Sep 17 18:58:50.603555 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.603443    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f90e64a-713f-470e-80bc-aad88bdebf84-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t\" (UID: \"3f90e64a-713f-470e-80bc-aad88bdebf84\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t"
Sep 17 18:58:50.603555 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.603469    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4xr2\" (UniqueName: \"kubernetes.io/projected/3f90e64a-713f-470e-80bc-aad88bdebf84-kube-api-access-p4xr2\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t\" (UID: \"3f90e64a-713f-470e-80bc-aad88bdebf84\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t"
Sep 17 18:58:50.603555 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.603514    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f90e64a-713f-470e-80bc-aad88bdebf84-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t\" (UID: \"3f90e64a-713f-470e-80bc-aad88bdebf84\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t"
Sep 17 18:58:50.603763 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.603593    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f90e64a-713f-470e-80bc-aad88bdebf84-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t\" (UID: \"3f90e64a-713f-470e-80bc-aad88bdebf84\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t"
Sep 17 18:58:50.603870 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.603848    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f90e64a-713f-470e-80bc-aad88bdebf84-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t\" (UID: \"3f90e64a-713f-470e-80bc-aad88bdebf84\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t"
Sep 17 18:58:50.603935 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.603896    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f90e64a-713f-470e-80bc-aad88bdebf84-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t\" (UID: \"3f90e64a-713f-470e-80bc-aad88bdebf84\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t"
Sep 17 18:58:50.603935 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.603923    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f90e64a-713f-470e-80bc-aad88bdebf84-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t\" (UID: \"3f90e64a-713f-470e-80bc-aad88bdebf84\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t"
Sep 17 18:58:50.605821 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.605786    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f90e64a-713f-470e-80bc-aad88bdebf84-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t\" (UID: \"3f90e64a-713f-470e-80bc-aad88bdebf84\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t"
Sep 17 18:58:50.606072 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.606057    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f90e64a-713f-470e-80bc-aad88bdebf84-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t\" (UID: \"3f90e64a-713f-470e-80bc-aad88bdebf84\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t"
Sep 17 18:58:50.633325 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.633292    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4xr2\" (UniqueName: \"kubernetes.io/projected/3f90e64a-713f-470e-80bc-aad88bdebf84-kube-api-access-p4xr2\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t\" (UID: \"3f90e64a-713f-470e-80bc-aad88bdebf84\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t"
Sep 17 18:58:50.795158 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.795070    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t"
Sep 17 18:58:50.873066 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.872472    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc"]
Sep 17 18:58:50.878269 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.878247    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc"
Sep 17 18:58:50.881939 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.881916    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-59xsz\""
Sep 17 18:58:50.893543 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.893507    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc"]
Sep 17 18:58:50.905116 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.905086    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24vg2\" (UniqueName: \"kubernetes.io/projected/ae4b6bcb-15d7-4c19-9f92-b6b05d918c11-kube-api-access-24vg2\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc\" (UID: \"ae4b6bcb-15d7-4c19-9f92-b6b05d918c11\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc"
Sep 17 18:58:50.905280 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.905192    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae4b6bcb-15d7-4c19-9f92-b6b05d918c11-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc\" (UID: \"ae4b6bcb-15d7-4c19-9f92-b6b05d918c11\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc"
Sep 17 18:58:50.911345 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:50.911322    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:58:51.006384 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:51.006343    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae4b6bcb-15d7-4c19-9f92-b6b05d918c11-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc\" (UID: \"ae4b6bcb-15d7-4c19-9f92-b6b05d918c11\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc"
Sep 17 18:58:51.006639 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:51.006395    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24vg2\" (UniqueName: \"kubernetes.io/projected/ae4b6bcb-15d7-4c19-9f92-b6b05d918c11-kube-api-access-24vg2\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc\" (UID: \"ae4b6bcb-15d7-4c19-9f92-b6b05d918c11\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc"
Sep 17 18:58:51.008922 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:51.008897    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae4b6bcb-15d7-4c19-9f92-b6b05d918c11-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc\" (UID: \"ae4b6bcb-15d7-4c19-9f92-b6b05d918c11\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc"
Sep 17 18:58:51.024564 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:51.024518    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24vg2\" (UniqueName: \"kubernetes.io/projected/ae4b6bcb-15d7-4c19-9f92-b6b05d918c11-kube-api-access-24vg2\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc\" (UID: \"ae4b6bcb-15d7-4c19-9f92-b6b05d918c11\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc"
Sep 17 18:58:51.083307 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:51.083276    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t"]
Sep 17 18:58:51.086837 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:58:51.086813    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f90e64a_713f_470e_80bc_aad88bdebf84.slice/crio-86c0aca3269ab23d0fe21f1dcff54f710d61bb51debac8103052895b7c7268e3 WatchSource:0}: Error finding container 86c0aca3269ab23d0fe21f1dcff54f710d61bb51debac8103052895b7c7268e3: Status 404 returned error can't find the container with id 86c0aca3269ab23d0fe21f1dcff54f710d61bb51debac8103052895b7c7268e3
Sep 17 18:58:51.191473 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:51.191441    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc"
Sep 17 18:58:51.473143 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:51.472972    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc"]
Sep 17 18:58:51.475729 ip-10-0-136-132 kubenswrapper[2588]: W0917 18:58:51.475702    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae4b6bcb_15d7_4c19_9f92_b6b05d918c11.slice/crio-673c93b49357445f0b86d7e32e40a1ef8aea2f53bea88482a1581605192a4d8b WatchSource:0}: Error finding container 673c93b49357445f0b86d7e32e40a1ef8aea2f53bea88482a1581605192a4d8b: Status 404 returned error can't find the container with id 673c93b49357445f0b86d7e32e40a1ef8aea2f53bea88482a1581605192a4d8b
Sep 17 18:58:51.966487 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:51.966441    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t" event={"ID":"3f90e64a-713f-470e-80bc-aad88bdebf84","Type":"ContainerStarted","Data":"77416df27201096c0378d752d3e760f0a865ea45695311263abb3553d03b4461"}
Sep 17 18:58:51.966958 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:51.966549    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t" event={"ID":"3f90e64a-713f-470e-80bc-aad88bdebf84","Type":"ContainerStarted","Data":"86c0aca3269ab23d0fe21f1dcff54f710d61bb51debac8103052895b7c7268e3"}
Sep 17 18:58:51.968376 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:51.968339    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc" event={"ID":"ae4b6bcb-15d7-4c19-9f92-b6b05d918c11","Type":"ContainerStarted","Data":"7fa1b1641d65a0d71f4e5f875484cc7b419d96b89f3c2c0452c0d3af02b8fa42"}
Sep 17 18:58:51.968546 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:51.968397    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc"
Sep 17 18:58:51.968546 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:51.968417    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc" event={"ID":"ae4b6bcb-15d7-4c19-9f92-b6b05d918c11","Type":"ContainerStarted","Data":"673c93b49357445f0b86d7e32e40a1ef8aea2f53bea88482a1581605192a4d8b"}
Sep 17 18:58:52.008006 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:52.007938    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t" podStartSLOduration=2.007914701 podStartE2EDuration="2.007914701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:58:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:58:52.006252379 +0000 UTC m=+1768.857257782" watchObservedRunningTime="2026-09-17 18:58:52.007914701 +0000 UTC m=+1768.858920101"
Sep 17 18:58:52.033241 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:52.033182    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc" podStartSLOduration=2.033165295 podStartE2EDuration="2.033165295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:58:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:58:52.031138065 +0000 UTC m=+1768.882143463" watchObservedRunningTime="2026-09-17 18:58:52.033165295 +0000 UTC m=+1768.884170694"
Sep 17 18:58:52.795379 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:52.795336    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t"
Sep 17 18:58:57.433507 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:58:57.433469    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:59:00.795558 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:59:00.795510    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t"
Sep 17 18:59:00.807829 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:59:00.807807    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t"
Sep 17 18:59:01.020707 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:59:01.020668    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t"
Sep 17 18:59:04.011240 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:59:04.011200    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:59:04.111984 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:59:04.111949    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:59:22.978113 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:59:22.978081    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc"
Sep 17 18:59:25.578340 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:59:25.578314    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xlklb_07ee31e0-43bb-4940-837d-969bd840492b/console-operator/1.log"
Sep 17 18:59:25.581122 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:59:25.581101    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xlklb_07ee31e0-43bb-4940-837d-969bd840492b/console-operator/1.log"
Sep 17 18:59:25.583798 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:59:25.583779    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovn-acl-logging/0.log"
Sep 17 18:59:25.586451 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:59:25.586433    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovn-acl-logging/0.log"
Sep 17 18:59:43.597362 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:59:43.597324    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 18:59:43.939593 ip-10-0-136-132 kubenswrapper[2588]: I0917 18:59:43.939504    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:03.512840 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:03.512808    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:04.161944 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:04.161911    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:06.326778 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:06.326747    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:18.062763 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:18.062730    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:18.162465 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:18.162437    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:35.662726 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:35.662686    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:40.626792 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:40.626758    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:40.727860 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:40.727819    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:45.574515 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:45.574479    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"]
Sep 17 19:00:45.574945 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:45.574697    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7" podUID="c92f25ed-3214-4ce8-9d43-cadf9064d08e" containerName="istio-proxy" containerID="cri-o://64ce808a47bf3d99aff38529954241010f5d8d8177d0a36291d3743cff18851b" gracePeriod=30
Sep 17 19:00:45.669938 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:45.669906    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:46.304312 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:46.304278    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:46.483567 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:46.483518    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt"]
Sep 17 19:00:46.487142 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:46.487117    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt"
Sep 17 19:00:46.490843 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:46.490816    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-nkns4\""
Sep 17 19:00:46.490996 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:46.490827    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 17 19:00:46.491076 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:46.490844    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 17 19:00:46.491134 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:46.490913    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-9rmwz\""
Sep 17 19:00:46.491134 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:46.490939    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 17 19:00:46.502545 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:46.502498    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt"]
Sep 17 19:00:46.531875 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:46.531850    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:46.658875 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:46.658785    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d828469c-7a91-4145-b90e-5afc139721f4-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt\" (UID: \"d828469c-7a91-4145-b90e-5afc139721f4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt"
Sep 17 19:00:46.659328 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:46.659033    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnsvk\" (UniqueName: \"kubernetes.io/projected/d828469c-7a91-4145-b90e-5afc139721f4-kube-api-access-vnsvk\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt\" (UID: \"d828469c-7a91-4145-b90e-5afc139721f4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt"
Sep 17 19:00:46.759708 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:46.759677    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnsvk\" (UniqueName: \"kubernetes.io/projected/d828469c-7a91-4145-b90e-5afc139721f4-kube-api-access-vnsvk\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt\" (UID: \"d828469c-7a91-4145-b90e-5afc139721f4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt"
Sep 17 19:00:46.759861 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:46.759726    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d828469c-7a91-4145-b90e-5afc139721f4-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt\" (UID: \"d828469c-7a91-4145-b90e-5afc139721f4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt"
Sep 17 19:00:46.762237 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:46.762218    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d828469c-7a91-4145-b90e-5afc139721f4-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt\" (UID: \"d828469c-7a91-4145-b90e-5afc139721f4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt"
Sep 17 19:00:46.768860 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:46.768836    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnsvk\" (UniqueName: \"kubernetes.io/projected/d828469c-7a91-4145-b90e-5afc139721f4-kube-api-access-vnsvk\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt\" (UID: \"d828469c-7a91-4145-b90e-5afc139721f4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt"
Sep 17 19:00:46.798479 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:46.798452    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt"
Sep 17 19:00:47.086818 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:47.086790    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt"]
Sep 17 19:00:47.089609 ip-10-0-136-132 kubenswrapper[2588]: W0917 19:00:47.089580    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd828469c_7a91_4145_b90e_5afc139721f4.slice/crio-a0ea55351fb2959f4b973a7424a3d9af1e46ca3de54a165d6b1e8bfb5eb53736 WatchSource:0}: Error finding container a0ea55351fb2959f4b973a7424a3d9af1e46ca3de54a165d6b1e8bfb5eb53736: Status 404 returned error can't find the container with id a0ea55351fb2959f4b973a7424a3d9af1e46ca3de54a165d6b1e8bfb5eb53736
Sep 17 19:00:47.477160 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:47.477066    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt" event={"ID":"d828469c-7a91-4145-b90e-5afc139721f4","Type":"ContainerStarted","Data":"81138c39fd3f0945e92f7b699d768d0c7b197ec69fb8c5f3b8c4318fc2b735c3"}
Sep 17 19:00:47.477429 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:47.477396    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt"
Sep 17 19:00:47.477582 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:47.477442    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt" event={"ID":"d828469c-7a91-4145-b90e-5afc139721f4","Type":"ContainerStarted","Data":"a0ea55351fb2959f4b973a7424a3d9af1e46ca3de54a165d6b1e8bfb5eb53736"}
Sep 17 19:00:47.681594 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:47.681557    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7" podUID="c92f25ed-3214-4ce8-9d43-cadf9064d08e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:00:50.829883 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:50.829856    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 19:00:50.859647 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:50.859561    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt" podStartSLOduration=4.8595196210000005 podStartE2EDuration="4.859519621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:00:47.497630339 +0000 UTC m=+1884.348635751" watchObservedRunningTime="2026-09-17 19:00:50.859519621 +0000 UTC m=+1887.710525021"
Sep 17 19:00:50.896010 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:50.895979    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c92f25ed-3214-4ce8-9d43-cadf9064d08e-istio-podinfo\") pod \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") "
Sep 17 19:00:50.896162 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:50.896070    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92f25ed-3214-4ce8-9d43-cadf9064d08e-workload-socket\") pod \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") "
Sep 17 19:00:50.896162 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:50.896098    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c92f25ed-3214-4ce8-9d43-cadf9064d08e-istio-envoy\") pod \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") "
Sep 17 19:00:50.896162 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:50.896121    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c92f25ed-3214-4ce8-9d43-cadf9064d08e-istiod-ca-cert\") pod \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") "
Sep 17 19:00:50.896162 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:50.896147    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92f25ed-3214-4ce8-9d43-cadf9064d08e-credential-socket\") pod \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") "
Sep 17 19:00:50.896409 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:50.896177    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgqfj\" (UniqueName: \"kubernetes.io/projected/c92f25ed-3214-4ce8-9d43-cadf9064d08e-kube-api-access-sgqfj\") pod \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") "
Sep 17 19:00:50.896409 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:50.896212    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c92f25ed-3214-4ce8-9d43-cadf9064d08e-istio-data\") pod \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") "
Sep 17 19:00:50.896409 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:50.896254    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c92f25ed-3214-4ce8-9d43-cadf9064d08e-workload-certs\") pod \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") "
Sep 17 19:00:50.896409 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:50.896286    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c92f25ed-3214-4ce8-9d43-cadf9064d08e-istio-token\") pod \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\" (UID: \"c92f25ed-3214-4ce8-9d43-cadf9064d08e\") "
Sep 17 19:00:50.897178 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:50.896946    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c92f25ed-3214-4ce8-9d43-cadf9064d08e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c92f25ed-3214-4ce8-9d43-cadf9064d08e" (UID: "c92f25ed-3214-4ce8-9d43-cadf9064d08e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:00:50.897178 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:50.897089    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c92f25ed-3214-4ce8-9d43-cadf9064d08e-istio-data" (OuterVolumeSpecName: "istio-data") pod "c92f25ed-3214-4ce8-9d43-cadf9064d08e" (UID: "c92f25ed-3214-4ce8-9d43-cadf9064d08e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:00:50.897178 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:50.897114    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c92f25ed-3214-4ce8-9d43-cadf9064d08e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c92f25ed-3214-4ce8-9d43-cadf9064d08e" (UID: "c92f25ed-3214-4ce8-9d43-cadf9064d08e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:00:50.897404 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:50.897349    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c92f25ed-3214-4ce8-9d43-cadf9064d08e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c92f25ed-3214-4ce8-9d43-cadf9064d08e" (UID: "c92f25ed-3214-4ce8-9d43-cadf9064d08e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:00:50.897812 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:50.897568    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c92f25ed-3214-4ce8-9d43-cadf9064d08e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c92f25ed-3214-4ce8-9d43-cadf9064d08e" (UID: "c92f25ed-3214-4ce8-9d43-cadf9064d08e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:00:50.899450 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:50.899369    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c92f25ed-3214-4ce8-9d43-cadf9064d08e-kube-api-access-sgqfj" (OuterVolumeSpecName: "kube-api-access-sgqfj") pod "c92f25ed-3214-4ce8-9d43-cadf9064d08e" (UID: "c92f25ed-3214-4ce8-9d43-cadf9064d08e"). InnerVolumeSpecName "kube-api-access-sgqfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:00:50.899450 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:50.899395    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c92f25ed-3214-4ce8-9d43-cadf9064d08e-istio-token" (OuterVolumeSpecName: "istio-token") pod "c92f25ed-3214-4ce8-9d43-cadf9064d08e" (UID: "c92f25ed-3214-4ce8-9d43-cadf9064d08e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:00:50.899651 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:50.899475    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c92f25ed-3214-4ce8-9d43-cadf9064d08e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c92f25ed-3214-4ce8-9d43-cadf9064d08e" (UID: "c92f25ed-3214-4ce8-9d43-cadf9064d08e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 19:00:50.899899 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:50.899866    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c92f25ed-3214-4ce8-9d43-cadf9064d08e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c92f25ed-3214-4ce8-9d43-cadf9064d08e" (UID: "c92f25ed-3214-4ce8-9d43-cadf9064d08e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:00:50.997033 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:50.997001    2588 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92f25ed-3214-4ce8-9d43-cadf9064d08e-workload-socket\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:00:50.997033 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:50.997025    2588 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c92f25ed-3214-4ce8-9d43-cadf9064d08e-istio-envoy\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:00:50.997033 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:50.997034    2588 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c92f25ed-3214-4ce8-9d43-cadf9064d08e-istiod-ca-cert\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:00:50.997033 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:50.997042    2588 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c92f25ed-3214-4ce8-9d43-cadf9064d08e-credential-socket\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:00:50.997289 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:50.997051    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgqfj\" (UniqueName: \"kubernetes.io/projected/c92f25ed-3214-4ce8-9d43-cadf9064d08e-kube-api-access-sgqfj\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:00:50.997289 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:50.997060    2588 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c92f25ed-3214-4ce8-9d43-cadf9064d08e-istio-data\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:00:50.997289 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:50.997068    2588 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c92f25ed-3214-4ce8-9d43-cadf9064d08e-workload-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:00:50.997289 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:50.997075    2588 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c92f25ed-3214-4ce8-9d43-cadf9064d08e-istio-token\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:00:50.997289 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:50.997083    2588 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c92f25ed-3214-4ce8-9d43-cadf9064d08e-istio-podinfo\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:00:51.496879 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:51.496845    2588 generic.go:356] "Generic (PLEG): container finished" podID="c92f25ed-3214-4ce8-9d43-cadf9064d08e" containerID="64ce808a47bf3d99aff38529954241010f5d8d8177d0a36291d3743cff18851b" exitCode=0
Sep 17 19:00:51.497084 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:51.496907    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7" event={"ID":"c92f25ed-3214-4ce8-9d43-cadf9064d08e","Type":"ContainerDied","Data":"64ce808a47bf3d99aff38529954241010f5d8d8177d0a36291d3743cff18851b"}
Sep 17 19:00:51.497084 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:51.496917    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"
Sep 17 19:00:51.497084 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:51.496950    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7" event={"ID":"c92f25ed-3214-4ce8-9d43-cadf9064d08e","Type":"ContainerDied","Data":"060a742e21c62a9101851766cef2874af8a95300cd6f191519fff1041adb320c"}
Sep 17 19:00:51.497084 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:51.496973    2588 scope.go:117] "RemoveContainer" containerID="64ce808a47bf3d99aff38529954241010f5d8d8177d0a36291d3743cff18851b"
Sep 17 19:00:51.506468 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:51.506452    2588 scope.go:117] "RemoveContainer" containerID="7616181c0f623af018fc474529e95b9ad87712158fcf1cc3b31ffe04aa2429d5"
Sep 17 19:00:51.513946 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:51.513932    2588 scope.go:117] "RemoveContainer" containerID="64ce808a47bf3d99aff38529954241010f5d8d8177d0a36291d3743cff18851b"
Sep 17 19:00:51.514199 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:00:51.514176    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64ce808a47bf3d99aff38529954241010f5d8d8177d0a36291d3743cff18851b\": container with ID starting with 64ce808a47bf3d99aff38529954241010f5d8d8177d0a36291d3743cff18851b not found: ID does not exist" containerID="64ce808a47bf3d99aff38529954241010f5d8d8177d0a36291d3743cff18851b"
Sep 17 19:00:51.514276 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:51.514209    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ce808a47bf3d99aff38529954241010f5d8d8177d0a36291d3743cff18851b"} err="failed to get container status \"64ce808a47bf3d99aff38529954241010f5d8d8177d0a36291d3743cff18851b\": rpc error: code = NotFound desc = could not find container \"64ce808a47bf3d99aff38529954241010f5d8d8177d0a36291d3743cff18851b\": container with ID starting with 64ce808a47bf3d99aff38529954241010f5d8d8177d0a36291d3743cff18851b not found: ID does not exist"
Sep 17 19:00:51.514276 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:51.514232    2588 scope.go:117] "RemoveContainer" containerID="7616181c0f623af018fc474529e95b9ad87712158fcf1cc3b31ffe04aa2429d5"
Sep 17 19:00:51.514511 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:00:51.514491    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7616181c0f623af018fc474529e95b9ad87712158fcf1cc3b31ffe04aa2429d5\": container with ID starting with 7616181c0f623af018fc474529e95b9ad87712158fcf1cc3b31ffe04aa2429d5 not found: ID does not exist" containerID="7616181c0f623af018fc474529e95b9ad87712158fcf1cc3b31ffe04aa2429d5"
Sep 17 19:00:51.514582 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:51.514517    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7616181c0f623af018fc474529e95b9ad87712158fcf1cc3b31ffe04aa2429d5"} err="failed to get container status \"7616181c0f623af018fc474529e95b9ad87712158fcf1cc3b31ffe04aa2429d5\": rpc error: code = NotFound desc = could not find container \"7616181c0f623af018fc474529e95b9ad87712158fcf1cc3b31ffe04aa2429d5\": container with ID starting with 7616181c0f623af018fc474529e95b9ad87712158fcf1cc3b31ffe04aa2429d5 not found: ID does not exist"
Sep 17 19:00:51.518059 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:51.518036    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"]
Sep 17 19:00:51.523887 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:51.523869    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8cd48b867-6rvm7"]
Sep 17 19:00:52.567097 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:52.567065    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:00:53.506639 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:00:53.506606    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c92f25ed-3214-4ce8-9d43-cadf9064d08e" path="/var/lib/kubelet/pods/c92f25ed-3214-4ce8-9d43-cadf9064d08e/volumes"
Sep 17 19:01:07.176327 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:07.176238    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:08.464620 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:08.464586    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:08.492569 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:08.492519    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t"]
Sep 17 19:01:08.501835 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:08.501802    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc"]
Sep 17 19:01:08.502501 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:01:08.502470    2588 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:01:08.502501 ip-10-0-136-132 kubenswrapper[2588]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:01:08.502501 ip-10-0-136-132 kubenswrapper[2588]:         , stdout: , stderr: , exit code -1
Sep 17 19:01:08.502501 ip-10-0-136-132 kubenswrapper[2588]:  > containerID="77416df27201096c0378d752d3e760f0a865ea45695311263abb3553d03b4461" cmd=["/bin/sleep","15"]
Sep 17 19:01:08.502790 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:08.502516    2588 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t" message="" err=<
Sep 17 19:01:08.502790 ip-10-0-136-132 kubenswrapper[2588]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:01:08.502790 ip-10-0-136-132 kubenswrapper[2588]:         , stdout: , stderr: , exit code -1
Sep 17 19:01:08.502790 ip-10-0-136-132 kubenswrapper[2588]:  >
Sep 17 19:01:08.502790 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:01:08.502553    2588 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:01:08.502790 ip-10-0-136-132 kubenswrapper[2588]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:01:08.502790 ip-10-0-136-132 kubenswrapper[2588]:         , stdout: , stderr: , exit code -1
Sep 17 19:01:08.502790 ip-10-0-136-132 kubenswrapper[2588]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t" podUID="3f90e64a-713f-470e-80bc-aad88bdebf84" containerName="main" containerID="cri-o://77416df27201096c0378d752d3e760f0a865ea45695311263abb3553d03b4461"
Sep 17 19:01:08.502790 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:08.502590    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t" podUID="3f90e64a-713f-470e-80bc-aad88bdebf84" containerName="main" containerID="cri-o://77416df27201096c0378d752d3e760f0a865ea45695311263abb3553d03b4461" gracePeriod=60
Sep 17 19:01:08.570658 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:08.570622    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:08.865403 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:08.865376    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t"
Sep 17 19:01:08.912072 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:08.909952    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:08.951770 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:08.951729    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f90e64a-713f-470e-80bc-aad88bdebf84-tmp-dir\") pod \"3f90e64a-713f-470e-80bc-aad88bdebf84\" (UID: \"3f90e64a-713f-470e-80bc-aad88bdebf84\") "
Sep 17 19:01:08.951939 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:08.951817    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f90e64a-713f-470e-80bc-aad88bdebf84-tls-certs\") pod \"3f90e64a-713f-470e-80bc-aad88bdebf84\" (UID: \"3f90e64a-713f-470e-80bc-aad88bdebf84\") "
Sep 17 19:01:08.951939 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:08.951875    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f90e64a-713f-470e-80bc-aad88bdebf84-dshm\") pod \"3f90e64a-713f-470e-80bc-aad88bdebf84\" (UID: \"3f90e64a-713f-470e-80bc-aad88bdebf84\") "
Sep 17 19:01:08.952061 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:08.951997    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f90e64a-713f-470e-80bc-aad88bdebf84-model-cache\") pod \"3f90e64a-713f-470e-80bc-aad88bdebf84\" (UID: \"3f90e64a-713f-470e-80bc-aad88bdebf84\") "
Sep 17 19:01:08.952118 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:08.952057    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4xr2\" (UniqueName: \"kubernetes.io/projected/3f90e64a-713f-470e-80bc-aad88bdebf84-kube-api-access-p4xr2\") pod \"3f90e64a-713f-470e-80bc-aad88bdebf84\" (UID: \"3f90e64a-713f-470e-80bc-aad88bdebf84\") "
Sep 17 19:01:08.952118 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:08.952005    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f90e64a-713f-470e-80bc-aad88bdebf84-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3f90e64a-713f-470e-80bc-aad88bdebf84" (UID: "3f90e64a-713f-470e-80bc-aad88bdebf84"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:01:08.952224 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:08.952110    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f90e64a-713f-470e-80bc-aad88bdebf84-home\") pod \"3f90e64a-713f-470e-80bc-aad88bdebf84\" (UID: \"3f90e64a-713f-470e-80bc-aad88bdebf84\") "
Sep 17 19:01:08.952224 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:08.952160    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f90e64a-713f-470e-80bc-aad88bdebf84-model-cache" (OuterVolumeSpecName: "model-cache") pod "3f90e64a-713f-470e-80bc-aad88bdebf84" (UID: "3f90e64a-713f-470e-80bc-aad88bdebf84"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:01:08.952335 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:08.952286    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f90e64a-713f-470e-80bc-aad88bdebf84-home" (OuterVolumeSpecName: "home") pod "3f90e64a-713f-470e-80bc-aad88bdebf84" (UID: "3f90e64a-713f-470e-80bc-aad88bdebf84"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:01:08.952459 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:08.952432    2588 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f90e64a-713f-470e-80bc-aad88bdebf84-tmp-dir\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:01:08.952550 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:08.952462    2588 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f90e64a-713f-470e-80bc-aad88bdebf84-model-cache\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:01:08.952550 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:08.952476    2588 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f90e64a-713f-470e-80bc-aad88bdebf84-home\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:01:08.954197 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:08.954173    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f90e64a-713f-470e-80bc-aad88bdebf84-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f90e64a-713f-470e-80bc-aad88bdebf84" (UID: "3f90e64a-713f-470e-80bc-aad88bdebf84"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:01:08.954197 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:08.954181    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f90e64a-713f-470e-80bc-aad88bdebf84-kube-api-access-p4xr2" (OuterVolumeSpecName: "kube-api-access-p4xr2") pod "3f90e64a-713f-470e-80bc-aad88bdebf84" (UID: "3f90e64a-713f-470e-80bc-aad88bdebf84"). InnerVolumeSpecName "kube-api-access-p4xr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:01:08.954410 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:08.954396    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f90e64a-713f-470e-80bc-aad88bdebf84-dshm" (OuterVolumeSpecName: "dshm") pod "3f90e64a-713f-470e-80bc-aad88bdebf84" (UID: "3f90e64a-713f-470e-80bc-aad88bdebf84"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:01:09.053031 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:09.053000    2588 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f90e64a-713f-470e-80bc-aad88bdebf84-dshm\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:01:09.053031 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:09.053032    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4xr2\" (UniqueName: \"kubernetes.io/projected/3f90e64a-713f-470e-80bc-aad88bdebf84-kube-api-access-p4xr2\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:01:09.053229 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:09.053044    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f90e64a-713f-470e-80bc-aad88bdebf84-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:01:09.580274 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:09.580236    2588 generic.go:356] "Generic (PLEG): container finished" podID="3f90e64a-713f-470e-80bc-aad88bdebf84" containerID="77416df27201096c0378d752d3e760f0a865ea45695311263abb3553d03b4461" exitCode=0
Sep 17 19:01:09.580784 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:09.580286    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t"
Sep 17 19:01:09.580784 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:09.580319    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t" event={"ID":"3f90e64a-713f-470e-80bc-aad88bdebf84","Type":"ContainerDied","Data":"77416df27201096c0378d752d3e760f0a865ea45695311263abb3553d03b4461"}
Sep 17 19:01:09.580784 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:09.580403    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t" event={"ID":"3f90e64a-713f-470e-80bc-aad88bdebf84","Type":"ContainerDied","Data":"86c0aca3269ab23d0fe21f1dcff54f710d61bb51debac8103052895b7c7268e3"}
Sep 17 19:01:09.580784 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:09.580434    2588 scope.go:117] "RemoveContainer" containerID="77416df27201096c0378d752d3e760f0a865ea45695311263abb3553d03b4461"
Sep 17 19:01:09.590375 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:09.590353    2588 scope.go:117] "RemoveContainer" containerID="77416df27201096c0378d752d3e760f0a865ea45695311263abb3553d03b4461"
Sep 17 19:01:09.590667 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:01:09.590648    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77416df27201096c0378d752d3e760f0a865ea45695311263abb3553d03b4461\": container with ID starting with 77416df27201096c0378d752d3e760f0a865ea45695311263abb3553d03b4461 not found: ID does not exist" containerID="77416df27201096c0378d752d3e760f0a865ea45695311263abb3553d03b4461"
Sep 17 19:01:09.590755 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:09.590680    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77416df27201096c0378d752d3e760f0a865ea45695311263abb3553d03b4461"} err="failed to get container status \"77416df27201096c0378d752d3e760f0a865ea45695311263abb3553d03b4461\": rpc error: code = NotFound desc = could not find container \"77416df27201096c0378d752d3e760f0a865ea45695311263abb3553d03b4461\": container with ID starting with 77416df27201096c0378d752d3e760f0a865ea45695311263abb3553d03b4461 not found: ID does not exist"
Sep 17 19:01:09.597482 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:09.597457    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t"]
Sep 17 19:01:09.608864 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:09.608840    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-r2c4t"]
Sep 17 19:01:11.505889 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:11.505855    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f90e64a-713f-470e-80bc-aad88bdebf84" path="/var/lib/kubelet/pods/3f90e64a-713f-470e-80bc-aad88bdebf84/volumes"
Sep 17 19:01:12.977463 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:12.977432    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc" podUID="ae4b6bcb-15d7-4c19-9f92-b6b05d918c11" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:01:14.757648 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:14.757617    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:16.085553 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:16.085510    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:19.488797 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:19.488771    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt"
Sep 17 19:01:22.978240 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:22.978205    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc" podUID="ae4b6bcb-15d7-4c19-9f92-b6b05d918c11" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:01:23.521007 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:23.520962    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc" podUID="ae4b6bcb-15d7-4c19-9f92-b6b05d918c11" containerName="main" containerID="cri-o://7fa1b1641d65a0d71f4e5f875484cc7b419d96b89f3c2c0452c0d3af02b8fa42" gracePeriod=45
Sep 17 19:01:29.059808 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:29.059770    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:29.620238 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:29.620204    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:29.696048 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:29.696016    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:32.978146 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:32.978111    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc" podUID="ae4b6bcb-15d7-4c19-9f92-b6b05d918c11" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:01:32.978571 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:32.978236    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc"
Sep 17 19:01:42.977696 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:42.977661    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc" podUID="ae4b6bcb-15d7-4c19-9f92-b6b05d918c11" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:01:52.977919 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:52.977881    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc" podUID="ae4b6bcb-15d7-4c19-9f92-b6b05d918c11" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:01:53.765089 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:53.765058    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc"
Sep 17 19:01:53.766374 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:53.766354    2588 generic.go:356] "Generic (PLEG): container finished" podID="ae4b6bcb-15d7-4c19-9f92-b6b05d918c11" containerID="7fa1b1641d65a0d71f4e5f875484cc7b419d96b89f3c2c0452c0d3af02b8fa42" exitCode=0
Sep 17 19:01:53.766468 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:53.766424    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc" event={"ID":"ae4b6bcb-15d7-4c19-9f92-b6b05d918c11","Type":"ContainerDied","Data":"7fa1b1641d65a0d71f4e5f875484cc7b419d96b89f3c2c0452c0d3af02b8fa42"}
Sep 17 19:01:53.766510 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:53.766476    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc" event={"ID":"ae4b6bcb-15d7-4c19-9f92-b6b05d918c11","Type":"ContainerDied","Data":"673c93b49357445f0b86d7e32e40a1ef8aea2f53bea88482a1581605192a4d8b"}
Sep 17 19:01:53.766510 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:53.766500    2588 scope.go:117] "RemoveContainer" containerID="7fa1b1641d65a0d71f4e5f875484cc7b419d96b89f3c2c0452c0d3af02b8fa42"
Sep 17 19:01:53.774082 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:53.774055    2588 scope.go:117] "RemoveContainer" containerID="7fa1b1641d65a0d71f4e5f875484cc7b419d96b89f3c2c0452c0d3af02b8fa42"
Sep 17 19:01:53.774341 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:01:53.774323    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fa1b1641d65a0d71f4e5f875484cc7b419d96b89f3c2c0452c0d3af02b8fa42\": container with ID starting with 7fa1b1641d65a0d71f4e5f875484cc7b419d96b89f3c2c0452c0d3af02b8fa42 not found: ID does not exist" containerID="7fa1b1641d65a0d71f4e5f875484cc7b419d96b89f3c2c0452c0d3af02b8fa42"
Sep 17 19:01:53.774400 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:53.774348    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fa1b1641d65a0d71f4e5f875484cc7b419d96b89f3c2c0452c0d3af02b8fa42"} err="failed to get container status \"7fa1b1641d65a0d71f4e5f875484cc7b419d96b89f3c2c0452c0d3af02b8fa42\": rpc error: code = NotFound desc = could not find container \"7fa1b1641d65a0d71f4e5f875484cc7b419d96b89f3c2c0452c0d3af02b8fa42\": container with ID starting with 7fa1b1641d65a0d71f4e5f875484cc7b419d96b89f3c2c0452c0d3af02b8fa42 not found: ID does not exist"
Sep 17 19:01:53.896242 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:53.896218    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24vg2\" (UniqueName: \"kubernetes.io/projected/ae4b6bcb-15d7-4c19-9f92-b6b05d918c11-kube-api-access-24vg2\") pod \"ae4b6bcb-15d7-4c19-9f92-b6b05d918c11\" (UID: \"ae4b6bcb-15d7-4c19-9f92-b6b05d918c11\") "
Sep 17 19:01:53.896443 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:53.896252    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae4b6bcb-15d7-4c19-9f92-b6b05d918c11-tls-certs\") pod \"ae4b6bcb-15d7-4c19-9f92-b6b05d918c11\" (UID: \"ae4b6bcb-15d7-4c19-9f92-b6b05d918c11\") "
Sep 17 19:01:53.898452 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:53.898417    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae4b6bcb-15d7-4c19-9f92-b6b05d918c11-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae4b6bcb-15d7-4c19-9f92-b6b05d918c11" (UID: "ae4b6bcb-15d7-4c19-9f92-b6b05d918c11"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:01:53.898452 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:53.898422    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae4b6bcb-15d7-4c19-9f92-b6b05d918c11-kube-api-access-24vg2" (OuterVolumeSpecName: "kube-api-access-24vg2") pod "ae4b6bcb-15d7-4c19-9f92-b6b05d918c11" (UID: "ae4b6bcb-15d7-4c19-9f92-b6b05d918c11"). InnerVolumeSpecName "kube-api-access-24vg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:01:53.997364 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:53.997339    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24vg2\" (UniqueName: \"kubernetes.io/projected/ae4b6bcb-15d7-4c19-9f92-b6b05d918c11-kube-api-access-24vg2\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:01:53.997364 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:53.997360    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae4b6bcb-15d7-4c19-9f92-b6b05d918c11-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:01:54.771841 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:54.771808    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc"
Sep 17 19:01:54.794916 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:54.794882    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc"]
Sep 17 19:01:54.803998 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:54.803979    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenpkbc"]
Sep 17 19:01:55.505568 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:55.505519    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae4b6bcb-15d7-4c19-9f92-b6b05d918c11" path="/var/lib/kubelet/pods/ae4b6bcb-15d7-4c19-9f92-b6b05d918c11/volumes"
Sep 17 19:01:58.338360 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:58.338321    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:01:58.434846 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:01:58.434819    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:02:04.850244 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:04.850209    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:02:07.138930 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.138728    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9"]
Sep 17 19:02:07.139352 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.139333    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae4b6bcb-15d7-4c19-9f92-b6b05d918c11" containerName="main"
Sep 17 19:02:07.139420 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.139354    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae4b6bcb-15d7-4c19-9f92-b6b05d918c11" containerName="main"
Sep 17 19:02:07.139420 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.139370    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c92f25ed-3214-4ce8-9d43-cadf9064d08e" containerName="istio-proxy"
Sep 17 19:02:07.139420 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.139379    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="c92f25ed-3214-4ce8-9d43-cadf9064d08e" containerName="istio-proxy"
Sep 17 19:02:07.139420 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.139397    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c92f25ed-3214-4ce8-9d43-cadf9064d08e" containerName="istio-proxy"
Sep 17 19:02:07.139420 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.139407    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="c92f25ed-3214-4ce8-9d43-cadf9064d08e" containerName="istio-proxy"
Sep 17 19:02:07.139420 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.139418    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f90e64a-713f-470e-80bc-aad88bdebf84" containerName="main"
Sep 17 19:02:07.139697 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.139427    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f90e64a-713f-470e-80bc-aad88bdebf84" containerName="main"
Sep 17 19:02:07.139697 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.139585    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f90e64a-713f-470e-80bc-aad88bdebf84" containerName="main"
Sep 17 19:02:07.139697 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.139598    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae4b6bcb-15d7-4c19-9f92-b6b05d918c11" containerName="main"
Sep 17 19:02:07.139697 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.139606    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="c92f25ed-3214-4ce8-9d43-cadf9064d08e" containerName="istio-proxy"
Sep 17 19:02:07.139697 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.139615    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="c92f25ed-3214-4ce8-9d43-cadf9064d08e" containerName="istio-proxy"
Sep 17 19:02:07.144971 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.144865    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9"
Sep 17 19:02:07.147923 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.147897    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 17 19:02:07.147923 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.147913    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 17 19:02:07.148844 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.148821    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 17 19:02:07.148955 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.148852    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-8m6kk\""
Sep 17 19:02:07.158404 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.158385    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9"]
Sep 17 19:02:07.161625 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.161605    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:02:07.189583 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.189560    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmgtv\" (UniqueName: \"kubernetes.io/projected/ec76caef-438c-4a89-92a6-e6e74261094a-kube-api-access-qmgtv\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9\" (UID: \"ec76caef-438c-4a89-92a6-e6e74261094a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9"
Sep 17 19:02:07.189683 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.189608    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec76caef-438c-4a89-92a6-e6e74261094a-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9\" (UID: \"ec76caef-438c-4a89-92a6-e6e74261094a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9"
Sep 17 19:02:07.189739 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.189694    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec76caef-438c-4a89-92a6-e6e74261094a-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9\" (UID: \"ec76caef-438c-4a89-92a6-e6e74261094a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9"
Sep 17 19:02:07.189739 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.189732    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec76caef-438c-4a89-92a6-e6e74261094a-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9\" (UID: \"ec76caef-438c-4a89-92a6-e6e74261094a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9"
Sep 17 19:02:07.189859 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.189761    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec76caef-438c-4a89-92a6-e6e74261094a-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9\" (UID: \"ec76caef-438c-4a89-92a6-e6e74261094a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9"
Sep 17 19:02:07.189859 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.189793    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec76caef-438c-4a89-92a6-e6e74261094a-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9\" (UID: \"ec76caef-438c-4a89-92a6-e6e74261094a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9"
Sep 17 19:02:07.291049 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.291002    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec76caef-438c-4a89-92a6-e6e74261094a-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9\" (UID: \"ec76caef-438c-4a89-92a6-e6e74261094a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9"
Sep 17 19:02:07.291049 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.291056    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec76caef-438c-4a89-92a6-e6e74261094a-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9\" (UID: \"ec76caef-438c-4a89-92a6-e6e74261094a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9"
Sep 17 19:02:07.291318 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.291083    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec76caef-438c-4a89-92a6-e6e74261094a-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9\" (UID: \"ec76caef-438c-4a89-92a6-e6e74261094a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9"
Sep 17 19:02:07.291318 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.291114    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec76caef-438c-4a89-92a6-e6e74261094a-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9\" (UID: \"ec76caef-438c-4a89-92a6-e6e74261094a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9"
Sep 17 19:02:07.291318 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.291133    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmgtv\" (UniqueName: \"kubernetes.io/projected/ec76caef-438c-4a89-92a6-e6e74261094a-kube-api-access-qmgtv\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9\" (UID: \"ec76caef-438c-4a89-92a6-e6e74261094a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9"
Sep 17 19:02:07.291318 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.291161    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec76caef-438c-4a89-92a6-e6e74261094a-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9\" (UID: \"ec76caef-438c-4a89-92a6-e6e74261094a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9"
Sep 17 19:02:07.291588 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.291563    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec76caef-438c-4a89-92a6-e6e74261094a-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9\" (UID: \"ec76caef-438c-4a89-92a6-e6e74261094a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9"
Sep 17 19:02:07.291588 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.291579    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec76caef-438c-4a89-92a6-e6e74261094a-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9\" (UID: \"ec76caef-438c-4a89-92a6-e6e74261094a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9"
Sep 17 19:02:07.291684 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.291642    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec76caef-438c-4a89-92a6-e6e74261094a-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9\" (UID: \"ec76caef-438c-4a89-92a6-e6e74261094a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9"
Sep 17 19:02:07.293464 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.293439    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec76caef-438c-4a89-92a6-e6e74261094a-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9\" (UID: \"ec76caef-438c-4a89-92a6-e6e74261094a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9"
Sep 17 19:02:07.293697 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.293681    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec76caef-438c-4a89-92a6-e6e74261094a-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9\" (UID: \"ec76caef-438c-4a89-92a6-e6e74261094a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9"
Sep 17 19:02:07.299973 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.299953    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmgtv\" (UniqueName: \"kubernetes.io/projected/ec76caef-438c-4a89-92a6-e6e74261094a-kube-api-access-qmgtv\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9\" (UID: \"ec76caef-438c-4a89-92a6-e6e74261094a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9"
Sep 17 19:02:07.455923 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.455854    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9"
Sep 17 19:02:07.486337 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.486312    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:02:07.731935 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.731907    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9"]
Sep 17 19:02:07.733747 ip-10-0-136-132 kubenswrapper[2588]: W0917 19:02:07.733712    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec76caef_438c_4a89_92a6_e6e74261094a.slice/crio-f2c7911d17ee913f7bba4a51aa5f0e7ecdaaeb90bd11bb9abec836cf792673c3 WatchSource:0}: Error finding container f2c7911d17ee913f7bba4a51aa5f0e7ecdaaeb90bd11bb9abec836cf792673c3: Status 404 returned error can't find the container with id f2c7911d17ee913f7bba4a51aa5f0e7ecdaaeb90bd11bb9abec836cf792673c3
Sep 17 19:02:07.824831 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.824795    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9" event={"ID":"ec76caef-438c-4a89-92a6-e6e74261094a","Type":"ContainerStarted","Data":"2ca406dcc63ddd851a960a9c6165243f70b42189f3a45769848fbe849a7dd525"}
Sep 17 19:02:07.824965 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.824855    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9" event={"ID":"ec76caef-438c-4a89-92a6-e6e74261094a","Type":"ContainerStarted","Data":"f2c7911d17ee913f7bba4a51aa5f0e7ecdaaeb90bd11bb9abec836cf792673c3"}
Sep 17 19:02:07.844557 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:07.844495    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9" podStartSLOduration=0.844480652 podStartE2EDuration="844.480652ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:02:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:02:07.843205691 +0000 UTC m=+1964.694211091" watchObservedRunningTime="2026-09-17 19:02:07.844480652 +0000 UTC m=+1964.695486051"
Sep 17 19:02:08.456434 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:08.456396    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9"
Sep 17 19:02:10.905646 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:10.905612    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:02:11.003988 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:11.003951    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:02:17.456126 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:17.456084    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9"
Sep 17 19:02:17.468673 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:17.468643    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9"
Sep 17 19:02:17.877389 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:17.877357    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9"
Sep 17 19:02:24.917159 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:24.917109    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:02:52.148864 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:52.148824    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:02:52.349851 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:52.349822    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m"]
Sep 17 19:02:52.352991 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:52.352970    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m"
Sep 17 19:02:52.355596 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:52.355571    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 17 19:02:52.356450 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:52.356436    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-bzcxf\""
Sep 17 19:02:52.370122 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:52.370098    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m"]
Sep 17 19:02:52.380205 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:52.380186    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:02:52.451370 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:52.451297    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b04807e9-6efc-420e-9e1c-6ca0ac78ad89-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m\" (UID: \"b04807e9-6efc-420e-9e1c-6ca0ac78ad89\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m"
Sep 17 19:02:52.451468 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:52.451370    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzl8n\" (UniqueName: \"kubernetes.io/projected/b04807e9-6efc-420e-9e1c-6ca0ac78ad89-kube-api-access-wzl8n\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m\" (UID: \"b04807e9-6efc-420e-9e1c-6ca0ac78ad89\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m"
Sep 17 19:02:52.551965 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:52.551933    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzl8n\" (UniqueName: \"kubernetes.io/projected/b04807e9-6efc-420e-9e1c-6ca0ac78ad89-kube-api-access-wzl8n\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m\" (UID: \"b04807e9-6efc-420e-9e1c-6ca0ac78ad89\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m"
Sep 17 19:02:52.552113 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:52.551996    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b04807e9-6efc-420e-9e1c-6ca0ac78ad89-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m\" (UID: \"b04807e9-6efc-420e-9e1c-6ca0ac78ad89\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m"
Sep 17 19:02:52.554434 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:52.554415    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b04807e9-6efc-420e-9e1c-6ca0ac78ad89-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m\" (UID: \"b04807e9-6efc-420e-9e1c-6ca0ac78ad89\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m"
Sep 17 19:02:52.564866 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:52.564841    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzl8n\" (UniqueName: \"kubernetes.io/projected/b04807e9-6efc-420e-9e1c-6ca0ac78ad89-kube-api-access-wzl8n\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m\" (UID: \"b04807e9-6efc-420e-9e1c-6ca0ac78ad89\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m"
Sep 17 19:02:52.663895 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:52.663867    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m"
Sep 17 19:02:52.947186 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:52.947107    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m"]
Sep 17 19:02:52.953127 ip-10-0-136-132 kubenswrapper[2588]: W0917 19:02:52.953100    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb04807e9_6efc_420e_9e1c_6ca0ac78ad89.slice/crio-8b70a132875fb68f194b2695631ca08a34b2adda971267bb85c295971983c65f WatchSource:0}: Error finding container 8b70a132875fb68f194b2695631ca08a34b2adda971267bb85c295971983c65f: Status 404 returned error can't find the container with id 8b70a132875fb68f194b2695631ca08a34b2adda971267bb85c295971983c65f
Sep 17 19:02:53.005156 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:53.005130    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m" event={"ID":"b04807e9-6efc-420e-9e1c-6ca0ac78ad89","Type":"ContainerStarted","Data":"8b70a132875fb68f194b2695631ca08a34b2adda971267bb85c295971983c65f"}
Sep 17 19:02:54.011189 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:54.011148    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m" event={"ID":"b04807e9-6efc-420e-9e1c-6ca0ac78ad89","Type":"ContainerStarted","Data":"f772da55ed61d8a475d307ca01aacfdeb5dee8155cffcf95bb4b966b87e09339"}
Sep 17 19:02:54.011729 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:54.011268    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m"
Sep 17 19:02:54.034439 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:02:54.034391    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m" podStartSLOduration=2.03437681 podStartE2EDuration="2.03437681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:02:54.031465865 +0000 UTC m=+2010.882471264" watchObservedRunningTime="2026-09-17 19:02:54.03437681 +0000 UTC m=+2010.885382199"
Sep 17 19:03:05.899697 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:03:05.899659    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:03:12.489493 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:03:12.489456    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:03:12.501003 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:03:12.500975    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt"]
Sep 17 19:03:12.567024 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:03:12.566996    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:03:19.488451 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:03:19.488413    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt" podUID="d828469c-7a91-4145-b90e-5afc139721f4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:03:25.017456 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:03:25.017424    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m"
Sep 17 19:03:26.622852 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:03:26.622821    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m"]
Sep 17 19:03:26.627737 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:03:26.627710    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:03:26.785181 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:03:26.785150    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:03:27.520455 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:03:27.520417    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt" podUID="d828469c-7a91-4145-b90e-5afc139721f4" containerName="main" containerID="cri-o://81138c39fd3f0945e92f7b699d768d0c7b197ec69fb8c5f3b8c4318fc2b735c3" gracePeriod=45
Sep 17 19:03:29.488817 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:03:29.488768    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt" podUID="d828469c-7a91-4145-b90e-5afc139721f4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:03:35.017569 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:03:35.017508    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m" podUID="b04807e9-6efc-420e-9e1c-6ca0ac78ad89" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:03:39.488561 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:03:39.488459    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt" podUID="d828469c-7a91-4145-b90e-5afc139721f4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:03:39.488901 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:03:39.488589    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt"
Sep 17 19:03:41.646239 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:03:41.646183    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m" podUID="b04807e9-6efc-420e-9e1c-6ca0ac78ad89" containerName="main" containerID="cri-o://f772da55ed61d8a475d307ca01aacfdeb5dee8155cffcf95bb4b966b87e09339" gracePeriod=45
Sep 17 19:03:45.017957 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:03:45.017921    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m" podUID="b04807e9-6efc-420e-9e1c-6ca0ac78ad89" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:03:49.488586 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:03:49.488549    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt" podUID="d828469c-7a91-4145-b90e-5afc139721f4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:03:55.017058 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:03:55.017022    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m" podUID="b04807e9-6efc-420e-9e1c-6ca0ac78ad89" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:03:55.017467 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:03:55.017125    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m"
Sep 17 19:03:57.762160 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:03:57.762134    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt"
Sep 17 19:03:57.889685 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:03:57.889657    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnsvk\" (UniqueName: \"kubernetes.io/projected/d828469c-7a91-4145-b90e-5afc139721f4-kube-api-access-vnsvk\") pod \"d828469c-7a91-4145-b90e-5afc139721f4\" (UID: \"d828469c-7a91-4145-b90e-5afc139721f4\") "
Sep 17 19:03:57.889864 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:03:57.889721    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d828469c-7a91-4145-b90e-5afc139721f4-tls-certs\") pod \"d828469c-7a91-4145-b90e-5afc139721f4\" (UID: \"d828469c-7a91-4145-b90e-5afc139721f4\") "
Sep 17 19:03:57.891941 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:03:57.891906    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d828469c-7a91-4145-b90e-5afc139721f4-kube-api-access-vnsvk" (OuterVolumeSpecName: "kube-api-access-vnsvk") pod "d828469c-7a91-4145-b90e-5afc139721f4" (UID: "d828469c-7a91-4145-b90e-5afc139721f4"). InnerVolumeSpecName "kube-api-access-vnsvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:03:57.892072 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:03:57.891960    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d828469c-7a91-4145-b90e-5afc139721f4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d828469c-7a91-4145-b90e-5afc139721f4" (UID: "d828469c-7a91-4145-b90e-5afc139721f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:03:57.991242 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:03:57.991207    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnsvk\" (UniqueName: \"kubernetes.io/projected/d828469c-7a91-4145-b90e-5afc139721f4-kube-api-access-vnsvk\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:03:57.991242 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:03:57.991237    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d828469c-7a91-4145-b90e-5afc139721f4-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:03:58.295979 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:03:58.295884    2588 generic.go:356] "Generic (PLEG): container finished" podID="d828469c-7a91-4145-b90e-5afc139721f4" containerID="81138c39fd3f0945e92f7b699d768d0c7b197ec69fb8c5f3b8c4318fc2b735c3" exitCode=0
Sep 17 19:03:58.295979 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:03:58.295932    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt"
Sep 17 19:03:58.295979 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:03:58.295963    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt" event={"ID":"d828469c-7a91-4145-b90e-5afc139721f4","Type":"ContainerDied","Data":"81138c39fd3f0945e92f7b699d768d0c7b197ec69fb8c5f3b8c4318fc2b735c3"}
Sep 17 19:03:58.296265 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:03:58.296013    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt" event={"ID":"d828469c-7a91-4145-b90e-5afc139721f4","Type":"ContainerDied","Data":"a0ea55351fb2959f4b973a7424a3d9af1e46ca3de54a165d6b1e8bfb5eb53736"}
Sep 17 19:03:58.296265 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:03:58.296037    2588 scope.go:117] "RemoveContainer" containerID="81138c39fd3f0945e92f7b699d768d0c7b197ec69fb8c5f3b8c4318fc2b735c3"
Sep 17 19:03:58.307512 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:03:58.307391    2588 scope.go:117] "RemoveContainer" containerID="81138c39fd3f0945e92f7b699d768d0c7b197ec69fb8c5f3b8c4318fc2b735c3"
Sep 17 19:03:58.307778 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:03:58.307756    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81138c39fd3f0945e92f7b699d768d0c7b197ec69fb8c5f3b8c4318fc2b735c3\": container with ID starting with 81138c39fd3f0945e92f7b699d768d0c7b197ec69fb8c5f3b8c4318fc2b735c3 not found: ID does not exist" containerID="81138c39fd3f0945e92f7b699d768d0c7b197ec69fb8c5f3b8c4318fc2b735c3"
Sep 17 19:03:58.307857 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:03:58.307784    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81138c39fd3f0945e92f7b699d768d0c7b197ec69fb8c5f3b8c4318fc2b735c3"} err="failed to get container status \"81138c39fd3f0945e92f7b699d768d0c7b197ec69fb8c5f3b8c4318fc2b735c3\": rpc error: code = NotFound desc = could not find container \"81138c39fd3f0945e92f7b699d768d0c7b197ec69fb8c5f3b8c4318fc2b735c3\": container with ID starting with 81138c39fd3f0945e92f7b699d768d0c7b197ec69fb8c5f3b8c4318fc2b735c3 not found: ID does not exist"
Sep 17 19:04:02.808696 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:02.808665    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt"]
Sep 17 19:04:02.813965 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:02.813938    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schegrklt"]
Sep 17 19:04:03.227318 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:03.227227    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:03.506374 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:03.506297    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d828469c-7a91-4145-b90e-5afc139721f4" path="/var/lib/kubelet/pods/d828469c-7a91-4145-b90e-5afc139721f4/volumes"
Sep 17 19:04:03.556675 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:03.556644    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:04.322026 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:04.321993    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:04.448162 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:04.448132    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:05.016988 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:05.016953    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m" podUID="b04807e9-6efc-420e-9e1c-6ca0ac78ad89" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:04:09.129796 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:09.129756    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:09.370016 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:09.369983    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd"]
Sep 17 19:04:09.370273 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:09.370261    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d828469c-7a91-4145-b90e-5afc139721f4" containerName="main"
Sep 17 19:04:09.370319 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:09.370276    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="d828469c-7a91-4145-b90e-5afc139721f4" containerName="main"
Sep 17 19:04:09.370354 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:09.370347    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="d828469c-7a91-4145-b90e-5afc139721f4" containerName="main"
Sep 17 19:04:09.374591 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:09.374573    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd"
Sep 17 19:04:09.379747 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:09.379712    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ac30d18-9199-4d3a-a704-a06f98739764-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd\" (UID: \"3ac30d18-9199-4d3a-a704-a06f98739764\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd"
Sep 17 19:04:09.379918 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:09.379793    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml4g2\" (UniqueName: \"kubernetes.io/projected/3ac30d18-9199-4d3a-a704-a06f98739764-kube-api-access-ml4g2\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd\" (UID: \"3ac30d18-9199-4d3a-a704-a06f98739764\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd"
Sep 17 19:04:09.382256 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:09.382233    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisvc-model-fb-opt-125m-oci-r-0fba19de-epp-sa-dockercfg-q9lpj\""
Sep 17 19:04:09.383233 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:09.383212    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 17 19:04:09.383393 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:09.383370    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-52l5t\""
Sep 17 19:04:09.383512 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:09.383427    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 17 19:04:09.383512 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:09.383315    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 17 19:04:09.395151 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:09.395127    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd"]
Sep 17 19:04:09.433481 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:09.433448    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:09.480966 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:09.480935    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ac30d18-9199-4d3a-a704-a06f98739764-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd\" (UID: \"3ac30d18-9199-4d3a-a704-a06f98739764\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd"
Sep 17 19:04:09.481315 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:09.481289    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml4g2\" (UniqueName: \"kubernetes.io/projected/3ac30d18-9199-4d3a-a704-a06f98739764-kube-api-access-ml4g2\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd\" (UID: \"3ac30d18-9199-4d3a-a704-a06f98739764\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd"
Sep 17 19:04:09.485552 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:09.484605    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ac30d18-9199-4d3a-a704-a06f98739764-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd\" (UID: \"3ac30d18-9199-4d3a-a704-a06f98739764\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd"
Sep 17 19:04:09.496284 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:09.496254    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml4g2\" (UniqueName: \"kubernetes.io/projected/3ac30d18-9199-4d3a-a704-a06f98739764-kube-api-access-ml4g2\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd\" (UID: \"3ac30d18-9199-4d3a-a704-a06f98739764\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd"
Sep 17 19:04:09.684814 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:09.684734    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd"
Sep 17 19:04:09.810259 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:09.810225    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:09.963618 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:09.963586    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd"]
Sep 17 19:04:09.965021 ip-10-0-136-132 kubenswrapper[2588]: W0917 19:04:09.964991    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac30d18_9199_4d3a_a704_a06f98739764.slice/crio-782df21f896a9e185e7f11adf444ec1168c769943f594968808551ee17407271 WatchSource:0}: Error finding container 782df21f896a9e185e7f11adf444ec1168c769943f594968808551ee17407271: Status 404 returned error can't find the container with id 782df21f896a9e185e7f11adf444ec1168c769943f594968808551ee17407271
Sep 17 19:04:10.351100 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:10.350819    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd" event={"ID":"3ac30d18-9199-4d3a-a704-a06f98739764","Type":"ContainerStarted","Data":"ac89d3ffe9fd64c4a235c7eb00bb0dcb7b802d309d8dab6413dc90c97d7e460c"}
Sep 17 19:04:10.351683 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:10.351661    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd"
Sep 17 19:04:10.351904 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:10.351886    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd" event={"ID":"3ac30d18-9199-4d3a-a704-a06f98739764","Type":"ContainerStarted","Data":"782df21f896a9e185e7f11adf444ec1168c769943f594968808551ee17407271"}
Sep 17 19:04:10.374828 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:10.374778    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd" podStartSLOduration=1.374762877 podStartE2EDuration="1.374762877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:04:10.373316034 +0000 UTC m=+2087.224321459" watchObservedRunningTime="2026-09-17 19:04:10.374762877 +0000 UTC m=+2087.225768276"
Sep 17 19:04:11.895741 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:11.895718    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m"
Sep 17 19:04:11.896720 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:11.896698    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzl8n\" (UniqueName: \"kubernetes.io/projected/b04807e9-6efc-420e-9e1c-6ca0ac78ad89-kube-api-access-wzl8n\") pod \"b04807e9-6efc-420e-9e1c-6ca0ac78ad89\" (UID: \"b04807e9-6efc-420e-9e1c-6ca0ac78ad89\") "
Sep 17 19:04:11.896835 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:11.896783    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b04807e9-6efc-420e-9e1c-6ca0ac78ad89-tls-certs\") pod \"b04807e9-6efc-420e-9e1c-6ca0ac78ad89\" (UID: \"b04807e9-6efc-420e-9e1c-6ca0ac78ad89\") "
Sep 17 19:04:11.898856 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:11.898828    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b04807e9-6efc-420e-9e1c-6ca0ac78ad89-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b04807e9-6efc-420e-9e1c-6ca0ac78ad89" (UID: "b04807e9-6efc-420e-9e1c-6ca0ac78ad89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:04:11.899009 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:11.898984    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b04807e9-6efc-420e-9e1c-6ca0ac78ad89-kube-api-access-wzl8n" (OuterVolumeSpecName: "kube-api-access-wzl8n") pod "b04807e9-6efc-420e-9e1c-6ca0ac78ad89" (UID: "b04807e9-6efc-420e-9e1c-6ca0ac78ad89"). InnerVolumeSpecName "kube-api-access-wzl8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:04:11.997249 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:11.997222    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzl8n\" (UniqueName: \"kubernetes.io/projected/b04807e9-6efc-420e-9e1c-6ca0ac78ad89-kube-api-access-wzl8n\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:04:11.997249 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:11.997246    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b04807e9-6efc-420e-9e1c-6ca0ac78ad89-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:04:12.362380 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:12.362352    2588 generic.go:356] "Generic (PLEG): container finished" podID="b04807e9-6efc-420e-9e1c-6ca0ac78ad89" containerID="f772da55ed61d8a475d307ca01aacfdeb5dee8155cffcf95bb4b966b87e09339" exitCode=0
Sep 17 19:04:12.362582 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:12.362396    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m"
Sep 17 19:04:12.362582 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:12.362440    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m" event={"ID":"b04807e9-6efc-420e-9e1c-6ca0ac78ad89","Type":"ContainerDied","Data":"f772da55ed61d8a475d307ca01aacfdeb5dee8155cffcf95bb4b966b87e09339"}
Sep 17 19:04:12.362582 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:12.362494    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m" event={"ID":"b04807e9-6efc-420e-9e1c-6ca0ac78ad89","Type":"ContainerDied","Data":"8b70a132875fb68f194b2695631ca08a34b2adda971267bb85c295971983c65f"}
Sep 17 19:04:12.362582 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:12.362522    2588 scope.go:117] "RemoveContainer" containerID="f772da55ed61d8a475d307ca01aacfdeb5dee8155cffcf95bb4b966b87e09339"
Sep 17 19:04:12.371378 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:12.371348    2588 scope.go:117] "RemoveContainer" containerID="f772da55ed61d8a475d307ca01aacfdeb5dee8155cffcf95bb4b966b87e09339"
Sep 17 19:04:12.371645 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:04:12.371624    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f772da55ed61d8a475d307ca01aacfdeb5dee8155cffcf95bb4b966b87e09339\": container with ID starting with f772da55ed61d8a475d307ca01aacfdeb5dee8155cffcf95bb4b966b87e09339 not found: ID does not exist" containerID="f772da55ed61d8a475d307ca01aacfdeb5dee8155cffcf95bb4b966b87e09339"
Sep 17 19:04:12.371715 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:12.371654    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f772da55ed61d8a475d307ca01aacfdeb5dee8155cffcf95bb4b966b87e09339"} err="failed to get container status \"f772da55ed61d8a475d307ca01aacfdeb5dee8155cffcf95bb4b966b87e09339\": rpc error: code = NotFound desc = could not find container \"f772da55ed61d8a475d307ca01aacfdeb5dee8155cffcf95bb4b966b87e09339\": container with ID starting with f772da55ed61d8a475d307ca01aacfdeb5dee8155cffcf95bb4b966b87e09339 not found: ID does not exist"
Sep 17 19:04:12.382931 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:12.382906    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m"]
Sep 17 19:04:12.387479 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:12.387458    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheldr5m"]
Sep 17 19:04:13.505301 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:13.505270    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b04807e9-6efc-420e-9e1c-6ca0ac78ad89" path="/var/lib/kubelet/pods/b04807e9-6efc-420e-9e1c-6ca0ac78ad89/volumes"
Sep 17 19:04:15.649035 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:15.648999    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:15.733026 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:15.732989    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:21.743381 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:21.743346    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:23.044668 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:23.044629    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9"]
Sep 17 19:04:23.046119 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:23.046090    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:23.053275 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:04:23.053242    2588 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:04:23.053275 ip-10-0-136-132 kubenswrapper[2588]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:04:23.053275 ip-10-0-136-132 kubenswrapper[2588]:         , stdout: , stderr: , exit code -1
Sep 17 19:04:23.053275 ip-10-0-136-132 kubenswrapper[2588]:  > containerID="2ca406dcc63ddd851a960a9c6165243f70b42189f3a45769848fbe849a7dd525" cmd=["/bin/sleep","15"]
Sep 17 19:04:23.053576 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:23.053300    2588 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9" message="" err=<
Sep 17 19:04:23.053576 ip-10-0-136-132 kubenswrapper[2588]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:04:23.053576 ip-10-0-136-132 kubenswrapper[2588]:         , stdout: , stderr: , exit code -1
Sep 17 19:04:23.053576 ip-10-0-136-132 kubenswrapper[2588]:  >
Sep 17 19:04:23.053576 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:04:23.053317    2588 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:04:23.053576 ip-10-0-136-132 kubenswrapper[2588]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:04:23.053576 ip-10-0-136-132 kubenswrapper[2588]:         , stdout: , stderr: , exit code -1
Sep 17 19:04:23.053576 ip-10-0-136-132 kubenswrapper[2588]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9" podUID="ec76caef-438c-4a89-92a6-e6e74261094a" containerName="main" containerID="cri-o://2ca406dcc63ddd851a960a9c6165243f70b42189f3a45769848fbe849a7dd525"
Sep 17 19:04:23.053576 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:23.053374    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9" podUID="ec76caef-438c-4a89-92a6-e6e74261094a" containerName="main" containerID="cri-o://2ca406dcc63ddd851a960a9c6165243f70b42189f3a45769848fbe849a7dd525" gracePeriod=60
Sep 17 19:04:23.141030 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:23.141002    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:23.388726 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:23.388700    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9"
Sep 17 19:04:23.417176 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:23.417149    2588 generic.go:356] "Generic (PLEG): container finished" podID="ec76caef-438c-4a89-92a6-e6e74261094a" containerID="2ca406dcc63ddd851a960a9c6165243f70b42189f3a45769848fbe849a7dd525" exitCode=0
Sep 17 19:04:23.417299 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:23.417208    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9"
Sep 17 19:04:23.417299 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:23.417210    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9" event={"ID":"ec76caef-438c-4a89-92a6-e6e74261094a","Type":"ContainerDied","Data":"2ca406dcc63ddd851a960a9c6165243f70b42189f3a45769848fbe849a7dd525"}
Sep 17 19:04:23.417377 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:23.417343    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9" event={"ID":"ec76caef-438c-4a89-92a6-e6e74261094a","Type":"ContainerDied","Data":"f2c7911d17ee913f7bba4a51aa5f0e7ecdaaeb90bd11bb9abec836cf792673c3"}
Sep 17 19:04:23.417377 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:23.417370    2588 scope.go:117] "RemoveContainer" containerID="2ca406dcc63ddd851a960a9c6165243f70b42189f3a45769848fbe849a7dd525"
Sep 17 19:04:23.425954 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:23.425936    2588 scope.go:117] "RemoveContainer" containerID="2ca406dcc63ddd851a960a9c6165243f70b42189f3a45769848fbe849a7dd525"
Sep 17 19:04:23.426217 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:04:23.426201    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ca406dcc63ddd851a960a9c6165243f70b42189f3a45769848fbe849a7dd525\": container with ID starting with 2ca406dcc63ddd851a960a9c6165243f70b42189f3a45769848fbe849a7dd525 not found: ID does not exist" containerID="2ca406dcc63ddd851a960a9c6165243f70b42189f3a45769848fbe849a7dd525"
Sep 17 19:04:23.426261 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:23.426224    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ca406dcc63ddd851a960a9c6165243f70b42189f3a45769848fbe849a7dd525"} err="failed to get container status \"2ca406dcc63ddd851a960a9c6165243f70b42189f3a45769848fbe849a7dd525\": rpc error: code = NotFound desc = could not find container \"2ca406dcc63ddd851a960a9c6165243f70b42189f3a45769848fbe849a7dd525\": container with ID starting with 2ca406dcc63ddd851a960a9c6165243f70b42189f3a45769848fbe849a7dd525 not found: ID does not exist"
Sep 17 19:04:23.485768 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:23.485744    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec76caef-438c-4a89-92a6-e6e74261094a-tls-certs\") pod \"ec76caef-438c-4a89-92a6-e6e74261094a\" (UID: \"ec76caef-438c-4a89-92a6-e6e74261094a\") "
Sep 17 19:04:23.485881 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:23.485806    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec76caef-438c-4a89-92a6-e6e74261094a-dshm\") pod \"ec76caef-438c-4a89-92a6-e6e74261094a\" (UID: \"ec76caef-438c-4a89-92a6-e6e74261094a\") "
Sep 17 19:04:23.485881 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:23.485824    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec76caef-438c-4a89-92a6-e6e74261094a-home\") pod \"ec76caef-438c-4a89-92a6-e6e74261094a\" (UID: \"ec76caef-438c-4a89-92a6-e6e74261094a\") "
Sep 17 19:04:23.485881 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:23.485852    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec76caef-438c-4a89-92a6-e6e74261094a-tmp-dir\") pod \"ec76caef-438c-4a89-92a6-e6e74261094a\" (UID: \"ec76caef-438c-4a89-92a6-e6e74261094a\") "
Sep 17 19:04:23.485881 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:23.485877    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmgtv\" (UniqueName: \"kubernetes.io/projected/ec76caef-438c-4a89-92a6-e6e74261094a-kube-api-access-qmgtv\") pod \"ec76caef-438c-4a89-92a6-e6e74261094a\" (UID: \"ec76caef-438c-4a89-92a6-e6e74261094a\") "
Sep 17 19:04:23.486114 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:23.485895    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec76caef-438c-4a89-92a6-e6e74261094a-model-cache\") pod \"ec76caef-438c-4a89-92a6-e6e74261094a\" (UID: \"ec76caef-438c-4a89-92a6-e6e74261094a\") "
Sep 17 19:04:23.486114 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:23.486063    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec76caef-438c-4a89-92a6-e6e74261094a-home" (OuterVolumeSpecName: "home") pod "ec76caef-438c-4a89-92a6-e6e74261094a" (UID: "ec76caef-438c-4a89-92a6-e6e74261094a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:04:23.486258 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:23.486130    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec76caef-438c-4a89-92a6-e6e74261094a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ec76caef-438c-4a89-92a6-e6e74261094a" (UID: "ec76caef-438c-4a89-92a6-e6e74261094a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:04:23.486258 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:23.486214    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec76caef-438c-4a89-92a6-e6e74261094a-model-cache" (OuterVolumeSpecName: "model-cache") pod "ec76caef-438c-4a89-92a6-e6e74261094a" (UID: "ec76caef-438c-4a89-92a6-e6e74261094a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:04:23.487893 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:23.487872    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec76caef-438c-4a89-92a6-e6e74261094a-dshm" (OuterVolumeSpecName: "dshm") pod "ec76caef-438c-4a89-92a6-e6e74261094a" (UID: "ec76caef-438c-4a89-92a6-e6e74261094a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:04:23.487982 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:23.487941    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec76caef-438c-4a89-92a6-e6e74261094a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ec76caef-438c-4a89-92a6-e6e74261094a" (UID: "ec76caef-438c-4a89-92a6-e6e74261094a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:04:23.488024 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:23.488010    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec76caef-438c-4a89-92a6-e6e74261094a-kube-api-access-qmgtv" (OuterVolumeSpecName: "kube-api-access-qmgtv") pod "ec76caef-438c-4a89-92a6-e6e74261094a" (UID: "ec76caef-438c-4a89-92a6-e6e74261094a"). InnerVolumeSpecName "kube-api-access-qmgtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:04:23.587025 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:23.586961    2588 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec76caef-438c-4a89-92a6-e6e74261094a-dshm\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:04:23.587025 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:23.586984    2588 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec76caef-438c-4a89-92a6-e6e74261094a-home\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:04:23.587025 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:23.586998    2588 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec76caef-438c-4a89-92a6-e6e74261094a-tmp-dir\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:04:23.587025 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:23.587012    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmgtv\" (UniqueName: \"kubernetes.io/projected/ec76caef-438c-4a89-92a6-e6e74261094a-kube-api-access-qmgtv\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:04:23.587025 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:23.587025    2588 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec76caef-438c-4a89-92a6-e6e74261094a-model-cache\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:04:23.587268 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:23.587038    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec76caef-438c-4a89-92a6-e6e74261094a-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:04:23.734862 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:23.734814    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9"]
Sep 17 19:04:23.739025 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:23.738997    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679697t4z9"]
Sep 17 19:04:25.505440 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:25.505408    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec76caef-438c-4a89-92a6-e6e74261094a" path="/var/lib/kubelet/pods/ec76caef-438c-4a89-92a6-e6e74261094a/volumes"
Sep 17 19:04:25.602268 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:25.602241    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xlklb_07ee31e0-43bb-4940-837d-969bd840492b/console-operator/1.log"
Sep 17 19:04:25.606155 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:25.606133    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xlklb_07ee31e0-43bb-4940-837d-969bd840492b/console-operator/1.log"
Sep 17 19:04:25.607744 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:25.607724    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovn-acl-logging/0.log"
Sep 17 19:04:25.611016 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:25.610997    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovn-acl-logging/0.log"
Sep 17 19:04:30.986056 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:30.986012    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:04:42.364417 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:04:42.364386    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd"
Sep 17 19:05:11.514866 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:11.514789    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:05:11.618149 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:11.618091    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:05:18.128639 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:18.128607    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:05:24.155793 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:24.155753    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:05:24.236263 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:24.236232    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:05:27.907875 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:27.907838    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"]
Sep 17 19:05:27.908250 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:27.908113    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec76caef-438c-4a89-92a6-e6e74261094a" containerName="main"
Sep 17 19:05:27.908250 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:27.908124    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec76caef-438c-4a89-92a6-e6e74261094a" containerName="main"
Sep 17 19:05:27.908250 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:27.908137    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b04807e9-6efc-420e-9e1c-6ca0ac78ad89" containerName="main"
Sep 17 19:05:27.908250 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:27.908143    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="b04807e9-6efc-420e-9e1c-6ca0ac78ad89" containerName="main"
Sep 17 19:05:27.908250 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:27.908193    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec76caef-438c-4a89-92a6-e6e74261094a" containerName="main"
Sep 17 19:05:27.908250 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:27.908199    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="b04807e9-6efc-420e-9e1c-6ca0ac78ad89" containerName="main"
Sep 17 19:05:27.911092 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:27.911072    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"
Sep 17 19:05:27.915176 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:27.915148    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Sep 17 19:05:27.915397 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:27.915372    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-jc9wl\""
Sep 17 19:05:27.915523 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:27.915191    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 17 19:05:27.915603 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:27.915304    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:05:27.915749 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:27.915731    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 17 19:05:27.915882 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:27.915867    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 17 19:05:27.926501 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:27.926472    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"]
Sep 17 19:05:27.951914 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:27.951884    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:05:28.002576 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:28.002508    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9\" (UID: \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"
Sep 17 19:05:28.002769 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:28.002595    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9\" (UID: \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"
Sep 17 19:05:28.002769 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:28.002637    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9\" (UID: \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"
Sep 17 19:05:28.002769 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:28.002689    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9\" (UID: \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"
Sep 17 19:05:28.002769 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:28.002717    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9\" (UID: \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"
Sep 17 19:05:28.002769 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:28.002736    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccclv\" (UniqueName: \"kubernetes.io/projected/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-kube-api-access-ccclv\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9\" (UID: \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"
Sep 17 19:05:28.002769 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:28.002757    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9\" (UID: \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"
Sep 17 19:05:28.002990 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:28.002779    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9\" (UID: \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"
Sep 17 19:05:28.103697 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:28.103654    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9\" (UID: \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"
Sep 17 19:05:28.103697 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:28.103704    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9\" (UID: \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"
Sep 17 19:05:28.103957 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:28.103737    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9\" (UID: \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"
Sep 17 19:05:28.103957 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:28.103762    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9\" (UID: \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"
Sep 17 19:05:28.103957 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:28.103779    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccclv\" (UniqueName: \"kubernetes.io/projected/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-kube-api-access-ccclv\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9\" (UID: \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"
Sep 17 19:05:28.103957 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:28.103804    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9\" (UID: \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"
Sep 17 19:05:28.103957 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:28.103836    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9\" (UID: \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"
Sep 17 19:05:28.103957 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:28.103866    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9\" (UID: \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"
Sep 17 19:05:28.104271 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:28.104123    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9\" (UID: \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"
Sep 17 19:05:28.104445 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:28.104418    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9\" (UID: \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"
Sep 17 19:05:28.104602 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:28.104555    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9\" (UID: \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"
Sep 17 19:05:28.104602 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:28.104592    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9\" (UID: \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"
Sep 17 19:05:28.104872 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:28.104846    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9\" (UID: \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"
Sep 17 19:05:28.106482 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:28.106462    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9\" (UID: \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"
Sep 17 19:05:28.106654 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:28.106631    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9\" (UID: \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"
Sep 17 19:05:28.113819 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:28.113797    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccclv\" (UniqueName: \"kubernetes.io/projected/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-kube-api-access-ccclv\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9\" (UID: \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"
Sep 17 19:05:28.222342 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:28.222233    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"
Sep 17 19:05:28.237079 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:28.237039    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:05:28.504460 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:28.504377    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"]
Sep 17 19:05:28.508012 ip-10-0-136-132 kubenswrapper[2588]: W0917 19:05:28.507978    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9ad0c03_5351_4d4a_a41b_8351ce3f21d7.slice/crio-0c15e9608cd0d33ae63f52abf6da66ba83e9d600d1abf2f9e64598e561653e4c WatchSource:0}: Error finding container 0c15e9608cd0d33ae63f52abf6da66ba83e9d600d1abf2f9e64598e561653e4c: Status 404 returned error can't find the container with id 0c15e9608cd0d33ae63f52abf6da66ba83e9d600d1abf2f9e64598e561653e4c
Sep 17 19:05:28.685610 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:28.685571    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9" event={"ID":"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7","Type":"ContainerStarted","Data":"60a158d508ffd6cb79f9599d025998f5034f6343fe9c571984dca550ea850b6e"}
Sep 17 19:05:28.685827 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:28.685622    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9" event={"ID":"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7","Type":"ContainerStarted","Data":"0c15e9608cd0d33ae63f52abf6da66ba83e9d600d1abf2f9e64598e561653e4c"}
Sep 17 19:05:31.698695 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:31.698608    2588 generic.go:356] "Generic (PLEG): container finished" podID="d9ad0c03-5351-4d4a-a41b-8351ce3f21d7" containerID="60a158d508ffd6cb79f9599d025998f5034f6343fe9c571984dca550ea850b6e" exitCode=0
Sep 17 19:05:31.698695 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:31.698680    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9" event={"ID":"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7","Type":"ContainerDied","Data":"60a158d508ffd6cb79f9599d025998f5034f6343fe9c571984dca550ea850b6e"}
Sep 17 19:05:32.705309 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:32.705275    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9" event={"ID":"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7","Type":"ContainerStarted","Data":"716209bc0c5521e2aef75c685b7f1912e5609791ae40d516b4b4d14c80ce0792"}
Sep 17 19:05:32.731635 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:32.731575    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9" podStartSLOduration=2.954256148 podStartE2EDuration="5.731554633s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.777298485s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:05:32.729137949 +0000 UTC m=+2169.580143341" watchObservedRunningTime="2026-09-17 19:05:32.731554633 +0000 UTC m=+2169.582560031"
Sep 17 19:05:32.921443 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:32.921416    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:05:33.222408 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:33.222363    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"
Sep 17 19:05:38.223023 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:38.222985    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"
Sep 17 19:05:38.224478 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:38.224448    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9" podUID="d9ad0c03-5351-4d4a-a41b-8351ce3f21d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 19:05:40.477571 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:40.477498    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:05:48.222788 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:48.222743    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9" podUID="d9ad0c03-5351-4d4a-a41b-8351ce3f21d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 19:05:58.222941 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:05:58.222901    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9" podUID="d9ad0c03-5351-4d4a-a41b-8351ce3f21d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 19:06:08.223209 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:06:08.223166    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9" podUID="d9ad0c03-5351-4d4a-a41b-8351ce3f21d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 19:06:14.148231 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:06:14.148198    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:14.154243 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:06:14.154210    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd"]
Sep 17 19:06:14.230824 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:06:14.230789    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:18.223095 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:06:18.223054    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9" podUID="d9ad0c03-5351-4d4a-a41b-8351ce3f21d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 19:06:22.364452 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:06:22.364411    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd" podUID="3ac30d18-9199-4d3a-a704-a06f98739764" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:06:28.223029 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:06:28.222979    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9" podUID="d9ad0c03-5351-4d4a-a41b-8351ce3f21d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 19:06:29.173631 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:06:29.173582    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd" podUID="3ac30d18-9199-4d3a-a704-a06f98739764" containerName="main" containerID="cri-o://ac89d3ffe9fd64c4a235c7eb00bb0dcb7b802d309d8dab6413dc90c97d7e460c" gracePeriod=45
Sep 17 19:06:31.043238 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:06:31.043208    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:32.364902 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:06:32.364865    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd" podUID="3ac30d18-9199-4d3a-a704-a06f98739764" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:06:38.223469 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:06:38.223420    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9" podUID="d9ad0c03-5351-4d4a-a41b-8351ce3f21d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 19:06:42.365001 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:06:42.364964    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd" podUID="3ac30d18-9199-4d3a-a704-a06f98739764" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:06:42.365451 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:06:42.365095    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd"
Sep 17 19:06:48.233295 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:06:48.233256    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"
Sep 17 19:06:48.240895 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:06:48.240864    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"
Sep 17 19:06:52.364767 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:06:52.364730    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd" podUID="3ac30d18-9199-4d3a-a704-a06f98739764" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:06:59.415987 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:06:59.415958    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd"
Sep 17 19:06:59.563248 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:06:59.563220    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ac30d18-9199-4d3a-a704-a06f98739764-tls-certs\") pod \"3ac30d18-9199-4d3a-a704-a06f98739764\" (UID: \"3ac30d18-9199-4d3a-a704-a06f98739764\") "
Sep 17 19:06:59.563436 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:06:59.563284    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ml4g2\" (UniqueName: \"kubernetes.io/projected/3ac30d18-9199-4d3a-a704-a06f98739764-kube-api-access-ml4g2\") pod \"3ac30d18-9199-4d3a-a704-a06f98739764\" (UID: \"3ac30d18-9199-4d3a-a704-a06f98739764\") "
Sep 17 19:06:59.565391 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:06:59.565360    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ac30d18-9199-4d3a-a704-a06f98739764-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3ac30d18-9199-4d3a-a704-a06f98739764" (UID: "3ac30d18-9199-4d3a-a704-a06f98739764"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:06:59.565502 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:06:59.565466    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac30d18-9199-4d3a-a704-a06f98739764-kube-api-access-ml4g2" (OuterVolumeSpecName: "kube-api-access-ml4g2") pod "3ac30d18-9199-4d3a-a704-a06f98739764" (UID: "3ac30d18-9199-4d3a-a704-a06f98739764"). InnerVolumeSpecName "kube-api-access-ml4g2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:06:59.664927 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:06:59.664885    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml4g2\" (UniqueName: \"kubernetes.io/projected/3ac30d18-9199-4d3a-a704-a06f98739764-kube-api-access-ml4g2\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:06:59.664927 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:06:59.664926    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ac30d18-9199-4d3a-a704-a06f98739764-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:07:00.084848 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:00.084817    2588 generic.go:356] "Generic (PLEG): container finished" podID="3ac30d18-9199-4d3a-a704-a06f98739764" containerID="ac89d3ffe9fd64c4a235c7eb00bb0dcb7b802d309d8dab6413dc90c97d7e460c" exitCode=0
Sep 17 19:07:00.085040 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:00.084902    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd"
Sep 17 19:07:00.085040 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:00.084908    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd" event={"ID":"3ac30d18-9199-4d3a-a704-a06f98739764","Type":"ContainerDied","Data":"ac89d3ffe9fd64c4a235c7eb00bb0dcb7b802d309d8dab6413dc90c97d7e460c"}
Sep 17 19:07:00.085040 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:00.084984    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd" event={"ID":"3ac30d18-9199-4d3a-a704-a06f98739764","Type":"ContainerDied","Data":"782df21f896a9e185e7f11adf444ec1168c769943f594968808551ee17407271"}
Sep 17 19:07:00.085040 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:00.085006    2588 scope.go:117] "RemoveContainer" containerID="ac89d3ffe9fd64c4a235c7eb00bb0dcb7b802d309d8dab6413dc90c97d7e460c"
Sep 17 19:07:00.093323 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:00.093304    2588 scope.go:117] "RemoveContainer" containerID="ac89d3ffe9fd64c4a235c7eb00bb0dcb7b802d309d8dab6413dc90c97d7e460c"
Sep 17 19:07:00.093640 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:07:00.093613    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac89d3ffe9fd64c4a235c7eb00bb0dcb7b802d309d8dab6413dc90c97d7e460c\": container with ID starting with ac89d3ffe9fd64c4a235c7eb00bb0dcb7b802d309d8dab6413dc90c97d7e460c not found: ID does not exist" containerID="ac89d3ffe9fd64c4a235c7eb00bb0dcb7b802d309d8dab6413dc90c97d7e460c"
Sep 17 19:07:00.093705 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:00.093652    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac89d3ffe9fd64c4a235c7eb00bb0dcb7b802d309d8dab6413dc90c97d7e460c"} err="failed to get container status \"ac89d3ffe9fd64c4a235c7eb00bb0dcb7b802d309d8dab6413dc90c97d7e460c\": rpc error: code = NotFound desc = could not find container \"ac89d3ffe9fd64c4a235c7eb00bb0dcb7b802d309d8dab6413dc90c97d7e460c\": container with ID starting with ac89d3ffe9fd64c4a235c7eb00bb0dcb7b802d309d8dab6413dc90c97d7e460c not found: ID does not exist"
Sep 17 19:07:00.125114 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:00.125084    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd"]
Sep 17 19:07:00.132714 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:00.132691    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schehlrwd"]
Sep 17 19:07:01.506000 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:01.505971    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ac30d18-9199-4d3a-a704-a06f98739764" path="/var/lib/kubelet/pods/3ac30d18-9199-4d3a-a704-a06f98739764/volumes"
Sep 17 19:07:04.088882 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:04.088850    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:04.186369 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:04.186319    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:04.283306 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:04.283265    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:10.478751 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:10.478717    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:10.902727 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:10.902692    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm"]
Sep 17 19:07:10.902988 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:10.902976    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ac30d18-9199-4d3a-a704-a06f98739764" containerName="main"
Sep 17 19:07:10.903038 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:10.902990    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ac30d18-9199-4d3a-a704-a06f98739764" containerName="main"
Sep 17 19:07:10.903073 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:10.903050    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ac30d18-9199-4d3a-a704-a06f98739764" containerName="main"
Sep 17 19:07:10.908378 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:10.908353    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm"
Sep 17 19:07:10.918132 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:10.918112    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 17 19:07:10.919134 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:10.919112    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-fcmxq\""
Sep 17 19:07:10.919226 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:10.919119    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 17 19:07:10.919226 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:10.919120    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 17 19:07:10.919312 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:10.919142    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-9tppj\""
Sep 17 19:07:10.958362 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:10.958324    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm"]
Sep 17 19:07:10.991179 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:10.991141    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:11.060455 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:11.060378    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5pr5\" (UniqueName: \"kubernetes.io/projected/2fcbbae6-f4a3-420e-a433-e446903ebd85-kube-api-access-f5pr5\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm\" (UID: \"2fcbbae6-f4a3-420e-a433-e446903ebd85\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm"
Sep 17 19:07:11.060699 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:11.060503    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fcbbae6-f4a3-420e-a433-e446903ebd85-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm\" (UID: \"2fcbbae6-f4a3-420e-a433-e446903ebd85\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm"
Sep 17 19:07:11.161803 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:11.161705    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fcbbae6-f4a3-420e-a433-e446903ebd85-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm\" (UID: \"2fcbbae6-f4a3-420e-a433-e446903ebd85\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm"
Sep 17 19:07:11.161948 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:11.161839    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5pr5\" (UniqueName: \"kubernetes.io/projected/2fcbbae6-f4a3-420e-a433-e446903ebd85-kube-api-access-f5pr5\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm\" (UID: \"2fcbbae6-f4a3-420e-a433-e446903ebd85\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm"
Sep 17 19:07:11.164237 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:11.164207    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fcbbae6-f4a3-420e-a433-e446903ebd85-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm\" (UID: \"2fcbbae6-f4a3-420e-a433-e446903ebd85\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm"
Sep 17 19:07:11.175031 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:11.175008    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5pr5\" (UniqueName: \"kubernetes.io/projected/2fcbbae6-f4a3-420e-a433-e446903ebd85-kube-api-access-f5pr5\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm\" (UID: \"2fcbbae6-f4a3-420e-a433-e446903ebd85\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm"
Sep 17 19:07:11.218703 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:11.218667    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm"
Sep 17 19:07:11.553626 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:11.553591    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm"]
Sep 17 19:07:11.558191 ip-10-0-136-132 kubenswrapper[2588]: W0917 19:07:11.558155    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fcbbae6_f4a3_420e_a433_e446903ebd85.slice/crio-3334604b7ab7c10d60df9ececf6a3158708d41e8b18e9b9df8d06cc763103d7a WatchSource:0}: Error finding container 3334604b7ab7c10d60df9ececf6a3158708d41e8b18e9b9df8d06cc763103d7a: Status 404 returned error can't find the container with id 3334604b7ab7c10d60df9ececf6a3158708d41e8b18e9b9df8d06cc763103d7a
Sep 17 19:07:12.145968 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:12.145923    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm" event={"ID":"2fcbbae6-f4a3-420e-a433-e446903ebd85","Type":"ContainerStarted","Data":"826f76e412cdc93a7fafbadc88cb90b8800f645bff0e7a2f0eaa2ea926dea50f"}
Sep 17 19:07:12.146152 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:12.145990    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm" event={"ID":"2fcbbae6-f4a3-420e-a433-e446903ebd85","Type":"ContainerStarted","Data":"3334604b7ab7c10d60df9ececf6a3158708d41e8b18e9b9df8d06cc763103d7a"}
Sep 17 19:07:12.146152 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:12.146039    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm"
Sep 17 19:07:12.246045 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:12.245990    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm" podStartSLOduration=2.245969368 podStartE2EDuration="2.245969368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:12.245378598 +0000 UTC m=+2269.096384001" watchObservedRunningTime="2026-09-17 19:07:12.245969368 +0000 UTC m=+2269.096974768"
Sep 17 19:07:27.454385 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:27.454351    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:31.385583 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:31.385543    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:32.808070 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:32.808038    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"]
Sep 17 19:07:32.860487 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:32.860455    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:32.958481 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:32.958447    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:34.340600 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:34.340496    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:34.512916 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:34.512883    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:43.152066 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:43.152037    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm"
Sep 17 19:07:47.836780 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:47.836723    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9" podUID="d9ad0c03-5351-4d4a-a41b-8351ce3f21d7" containerName="main" containerID="cri-o://716209bc0c5521e2aef75c685b7f1912e5609791ae40d516b4b4d14c80ce0792" gracePeriod=45
Sep 17 19:07:48.234567 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:48.234451    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9" podUID="d9ad0c03-5351-4d4a-a41b-8351ce3f21d7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 19:07:48.485405 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:48.485347    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"
Sep 17 19:07:48.576360 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:48.576321    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-home\") pod \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\" (UID: \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\") "
Sep 17 19:07:48.576360 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:48.576360    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-dshm\") pod \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\" (UID: \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\") "
Sep 17 19:07:48.576637 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:48.576409    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-tmp-dir\") pod \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\" (UID: \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\") "
Sep 17 19:07:48.576637 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:48.576444    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-kserve-provision-location\") pod \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\" (UID: \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\") "
Sep 17 19:07:48.576637 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:48.576467    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-tls-certs\") pod \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\" (UID: \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\") "
Sep 17 19:07:48.576637 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:48.576491    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccclv\" (UniqueName: \"kubernetes.io/projected/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-kube-api-access-ccclv\") pod \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\" (UID: \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\") "
Sep 17 19:07:48.576637 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:48.576521    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-model-cache\") pod \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\" (UID: \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\") "
Sep 17 19:07:48.576637 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:48.576586    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-cabundle-cert\") pod \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\" (UID: \"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7\") "
Sep 17 19:07:48.576963 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:48.576779    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-home" (OuterVolumeSpecName: "home") pod "d9ad0c03-5351-4d4a-a41b-8351ce3f21d7" (UID: "d9ad0c03-5351-4d4a-a41b-8351ce3f21d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:07:48.576963 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:48.576879    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-model-cache" (OuterVolumeSpecName: "model-cache") pod "d9ad0c03-5351-4d4a-a41b-8351ce3f21d7" (UID: "d9ad0c03-5351-4d4a-a41b-8351ce3f21d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:07:48.577156 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:48.577132    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d9ad0c03-5351-4d4a-a41b-8351ce3f21d7" (UID: "d9ad0c03-5351-4d4a-a41b-8351ce3f21d7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:07:48.579074 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:48.579051    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-kube-api-access-ccclv" (OuterVolumeSpecName: "kube-api-access-ccclv") pod "d9ad0c03-5351-4d4a-a41b-8351ce3f21d7" (UID: "d9ad0c03-5351-4d4a-a41b-8351ce3f21d7"). InnerVolumeSpecName "kube-api-access-ccclv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:07:48.579225 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:48.579206    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d9ad0c03-5351-4d4a-a41b-8351ce3f21d7" (UID: "d9ad0c03-5351-4d4a-a41b-8351ce3f21d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:07:48.579293 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:48.579213    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-dshm" (OuterVolumeSpecName: "dshm") pod "d9ad0c03-5351-4d4a-a41b-8351ce3f21d7" (UID: "d9ad0c03-5351-4d4a-a41b-8351ce3f21d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:07:48.587809 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:48.587783    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d9ad0c03-5351-4d4a-a41b-8351ce3f21d7" (UID: "d9ad0c03-5351-4d4a-a41b-8351ce3f21d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:07:48.604962 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:48.604933    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d9ad0c03-5351-4d4a-a41b-8351ce3f21d7" (UID: "d9ad0c03-5351-4d4a-a41b-8351ce3f21d7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:07:48.677772 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:48.677729    2588 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-dshm\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:07:48.677772 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:48.677763    2588 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-tmp-dir\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:07:48.677772 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:48.677774    2588 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-kserve-provision-location\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:07:48.677772 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:48.677783    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:07:48.678040 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:48.677792    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccclv\" (UniqueName: \"kubernetes.io/projected/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-kube-api-access-ccclv\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:07:48.678040 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:48.677800    2588 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-model-cache\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:07:48.678040 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:48.677809    2588 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-cabundle-cert\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:07:48.678040 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:48.677817    2588 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7-home\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:07:49.295826 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:49.295781    2588 generic.go:356] "Generic (PLEG): container finished" podID="d9ad0c03-5351-4d4a-a41b-8351ce3f21d7" containerID="716209bc0c5521e2aef75c685b7f1912e5609791ae40d516b4b4d14c80ce0792" exitCode=0
Sep 17 19:07:49.296296 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:49.295837    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"
Sep 17 19:07:49.296296 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:49.295870    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9" event={"ID":"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7","Type":"ContainerDied","Data":"716209bc0c5521e2aef75c685b7f1912e5609791ae40d516b4b4d14c80ce0792"}
Sep 17 19:07:49.296296 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:49.295926    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9" event={"ID":"d9ad0c03-5351-4d4a-a41b-8351ce3f21d7","Type":"ContainerDied","Data":"0c15e9608cd0d33ae63f52abf6da66ba83e9d600d1abf2f9e64598e561653e4c"}
Sep 17 19:07:49.296296 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:49.295955    2588 scope.go:117] "RemoveContainer" containerID="716209bc0c5521e2aef75c685b7f1912e5609791ae40d516b4b4d14c80ce0792"
Sep 17 19:07:49.304635 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:49.304617    2588 scope.go:117] "RemoveContainer" containerID="60a158d508ffd6cb79f9599d025998f5034f6343fe9c571984dca550ea850b6e"
Sep 17 19:07:49.312299 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:49.312281    2588 scope.go:117] "RemoveContainer" containerID="716209bc0c5521e2aef75c685b7f1912e5609791ae40d516b4b4d14c80ce0792"
Sep 17 19:07:49.312568 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:07:49.312550    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"716209bc0c5521e2aef75c685b7f1912e5609791ae40d516b4b4d14c80ce0792\": container with ID starting with 716209bc0c5521e2aef75c685b7f1912e5609791ae40d516b4b4d14c80ce0792 not found: ID does not exist" containerID="716209bc0c5521e2aef75c685b7f1912e5609791ae40d516b4b4d14c80ce0792"
Sep 17 19:07:49.312625 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:49.312575    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716209bc0c5521e2aef75c685b7f1912e5609791ae40d516b4b4d14c80ce0792"} err="failed to get container status \"716209bc0c5521e2aef75c685b7f1912e5609791ae40d516b4b4d14c80ce0792\": rpc error: code = NotFound desc = could not find container \"716209bc0c5521e2aef75c685b7f1912e5609791ae40d516b4b4d14c80ce0792\": container with ID starting with 716209bc0c5521e2aef75c685b7f1912e5609791ae40d516b4b4d14c80ce0792 not found: ID does not exist"
Sep 17 19:07:49.312625 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:49.312594    2588 scope.go:117] "RemoveContainer" containerID="60a158d508ffd6cb79f9599d025998f5034f6343fe9c571984dca550ea850b6e"
Sep 17 19:07:49.312850 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:07:49.312832    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60a158d508ffd6cb79f9599d025998f5034f6343fe9c571984dca550ea850b6e\": container with ID starting with 60a158d508ffd6cb79f9599d025998f5034f6343fe9c571984dca550ea850b6e not found: ID does not exist" containerID="60a158d508ffd6cb79f9599d025998f5034f6343fe9c571984dca550ea850b6e"
Sep 17 19:07:49.312899 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:49.312859    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60a158d508ffd6cb79f9599d025998f5034f6343fe9c571984dca550ea850b6e"} err="failed to get container status \"60a158d508ffd6cb79f9599d025998f5034f6343fe9c571984dca550ea850b6e\": rpc error: code = NotFound desc = could not find container \"60a158d508ffd6cb79f9599d025998f5034f6343fe9c571984dca550ea850b6e\": container with ID starting with 60a158d508ffd6cb79f9599d025998f5034f6343fe9c571984dca550ea850b6e not found: ID does not exist"
Sep 17 19:07:49.318926 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:49.318906    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"]
Sep 17 19:07:49.324952 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:49.324932    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-cf894795c-h48q9"]
Sep 17 19:07:49.505973 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:07:49.505943    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9ad0c03-5351-4d4a-a41b-8351ce3f21d7" path="/var/lib/kubelet/pods/d9ad0c03-5351-4d4a-a41b-8351ce3f21d7/volumes"
Sep 17 19:08:20.828638 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:20.828607    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:20.951121 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:20.951090    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:27.018383 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:27.018348    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:32.569725 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:32.569691    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:32.650788 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:32.650741    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:32.825085 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:32.824994    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:33.128797 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:33.128711    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl"]
Sep 17 19:08:33.129006 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:33.128994    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9ad0c03-5351-4d4a-a41b-8351ce3f21d7" containerName="storage-initializer"
Sep 17 19:08:33.129059 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:33.129007    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9ad0c03-5351-4d4a-a41b-8351ce3f21d7" containerName="storage-initializer"
Sep 17 19:08:33.129059 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:33.129021    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9ad0c03-5351-4d4a-a41b-8351ce3f21d7" containerName="main"
Sep 17 19:08:33.129059 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:33.129026    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9ad0c03-5351-4d4a-a41b-8351ce3f21d7" containerName="main"
Sep 17 19:08:33.129156 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:33.129091    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9ad0c03-5351-4d4a-a41b-8351ce3f21d7" containerName="main"
Sep 17 19:08:33.131884 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:33.131848    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl"
Sep 17 19:08:33.153359 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:33.153336    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-dtdtq\""
Sep 17 19:08:33.154682 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:33.154665    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 17 19:08:33.154935 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:33.154907    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-2kzz9\""
Sep 17 19:08:33.154935 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:33.154919    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 17 19:08:33.155062 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:33.154910    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 17 19:08:33.161671 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:33.161650    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl"]
Sep 17 19:08:33.211218 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:33.211188    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:33.227364 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:33.227334    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhvxl\" (UniqueName: \"kubernetes.io/projected/e247038a-662a-4efc-bbbb-2faa9f0976e2-kube-api-access-dhvxl\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl\" (UID: \"e247038a-662a-4efc-bbbb-2faa9f0976e2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl"
Sep 17 19:08:33.227487 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:33.227371    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e247038a-662a-4efc-bbbb-2faa9f0976e2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl\" (UID: \"e247038a-662a-4efc-bbbb-2faa9f0976e2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl"
Sep 17 19:08:33.327905 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:33.327870    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e247038a-662a-4efc-bbbb-2faa9f0976e2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl\" (UID: \"e247038a-662a-4efc-bbbb-2faa9f0976e2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl"
Sep 17 19:08:33.328061 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:33.327939    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhvxl\" (UniqueName: \"kubernetes.io/projected/e247038a-662a-4efc-bbbb-2faa9f0976e2-kube-api-access-dhvxl\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl\" (UID: \"e247038a-662a-4efc-bbbb-2faa9f0976e2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl"
Sep 17 19:08:33.330423 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:33.330402    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e247038a-662a-4efc-bbbb-2faa9f0976e2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl\" (UID: \"e247038a-662a-4efc-bbbb-2faa9f0976e2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl"
Sep 17 19:08:33.337133 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:33.337109    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhvxl\" (UniqueName: \"kubernetes.io/projected/e247038a-662a-4efc-bbbb-2faa9f0976e2-kube-api-access-dhvxl\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl\" (UID: \"e247038a-662a-4efc-bbbb-2faa9f0976e2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl"
Sep 17 19:08:33.441582 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:33.441453    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl"
Sep 17 19:08:33.719967 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:33.719858    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl"]
Sep 17 19:08:33.722833 ip-10-0-136-132 kubenswrapper[2588]: W0917 19:08:33.722797    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode247038a_662a_4efc_bbbb_2faa9f0976e2.slice/crio-a3b865dbf6cdd5ebbdc79d00fa1c2dbad5355be53a49ffc07bb2c38debbf584c WatchSource:0}: Error finding container a3b865dbf6cdd5ebbdc79d00fa1c2dbad5355be53a49ffc07bb2c38debbf584c: Status 404 returned error can't find the container with id a3b865dbf6cdd5ebbdc79d00fa1c2dbad5355be53a49ffc07bb2c38debbf584c
Sep 17 19:08:34.470657 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:34.470616    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl" event={"ID":"e247038a-662a-4efc-bbbb-2faa9f0976e2","Type":"ContainerStarted","Data":"7c6dccf0234e207d7a62383415c355a224ecb559e37cb0f1165cedd47edd394f"}
Sep 17 19:08:34.470843 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:34.470698    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl" event={"ID":"e247038a-662a-4efc-bbbb-2faa9f0976e2","Type":"ContainerStarted","Data":"a3b865dbf6cdd5ebbdc79d00fa1c2dbad5355be53a49ffc07bb2c38debbf584c"}
Sep 17 19:08:34.470843 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:34.470725    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl"
Sep 17 19:08:34.498198 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:34.498146    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl" podStartSLOduration=1.498132957 podStartE2EDuration="1.498132957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:08:34.497843572 +0000 UTC m=+2351.348848970" watchObservedRunningTime="2026-09-17 19:08:34.498132957 +0000 UTC m=+2351.349138357"
Sep 17 19:08:34.521649 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:34.521627    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:34.659597 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:34.659568    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:47.959983 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:08:47.959945    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:05.476603 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:09:05.476571    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl"
Sep 17 19:09:15.529662 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:09:15.529588    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm"]
Sep 17 19:09:15.557308 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:09:15.557259    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:15.654994 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:09:15.654943    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:23.151878 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:09:23.151838    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm" podUID="2fcbbae6-f4a3-420e-a433-e446903ebd85" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:09:25.626606 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:09:25.626576    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xlklb_07ee31e0-43bb-4940-837d-969bd840492b/console-operator/1.log"
Sep 17 19:09:25.630695 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:09:25.630673    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xlklb_07ee31e0-43bb-4940-837d-969bd840492b/console-operator/1.log"
Sep 17 19:09:25.631950 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:09:25.631931    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovn-acl-logging/0.log"
Sep 17 19:09:25.636111 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:09:25.636095    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovn-acl-logging/0.log"
Sep 17 19:09:30.549227 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:09:30.549156    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm" podUID="2fcbbae6-f4a3-420e-a433-e446903ebd85" containerName="main" containerID="cri-o://826f76e412cdc93a7fafbadc88cb90b8800f645bff0e7a2f0eaa2ea926dea50f" gracePeriod=45
Sep 17 19:09:33.152111 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:09:33.152069    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm" podUID="2fcbbae6-f4a3-420e-a433-e446903ebd85" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:09:43.152066 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:09:43.152024    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm" podUID="2fcbbae6-f4a3-420e-a433-e446903ebd85" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:09:43.152447 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:09:43.152158    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm"
Sep 17 19:09:53.152443 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:09:53.152409    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm" podUID="2fcbbae6-f4a3-420e-a433-e446903ebd85" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:09:54.359435 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:09:54.359387    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:54.359885 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:09:54.359835    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl"]
Sep 17 19:09:54.462241 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:09:54.462206    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:55.475652 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:09:55.475615    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl" podUID="e247038a-662a-4efc-bbbb-2faa9f0976e2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:10:00.786205 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:00.786180    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm"
Sep 17 19:10:00.812568 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:00.812495    2588 generic.go:356] "Generic (PLEG): container finished" podID="2fcbbae6-f4a3-420e-a433-e446903ebd85" containerID="826f76e412cdc93a7fafbadc88cb90b8800f645bff0e7a2f0eaa2ea926dea50f" exitCode=0
Sep 17 19:10:00.812684 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:00.812563    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm"
Sep 17 19:10:00.812684 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:00.812561    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm" event={"ID":"2fcbbae6-f4a3-420e-a433-e446903ebd85","Type":"ContainerDied","Data":"826f76e412cdc93a7fafbadc88cb90b8800f645bff0e7a2f0eaa2ea926dea50f"}
Sep 17 19:10:00.812760 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:00.812687    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm" event={"ID":"2fcbbae6-f4a3-420e-a433-e446903ebd85","Type":"ContainerDied","Data":"3334604b7ab7c10d60df9ececf6a3158708d41e8b18e9b9df8d06cc763103d7a"}
Sep 17 19:10:00.812760 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:00.812716    2588 scope.go:117] "RemoveContainer" containerID="826f76e412cdc93a7fafbadc88cb90b8800f645bff0e7a2f0eaa2ea926dea50f"
Sep 17 19:10:00.822043 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:00.822027    2588 scope.go:117] "RemoveContainer" containerID="826f76e412cdc93a7fafbadc88cb90b8800f645bff0e7a2f0eaa2ea926dea50f"
Sep 17 19:10:00.822294 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:10:00.822277    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"826f76e412cdc93a7fafbadc88cb90b8800f645bff0e7a2f0eaa2ea926dea50f\": container with ID starting with 826f76e412cdc93a7fafbadc88cb90b8800f645bff0e7a2f0eaa2ea926dea50f not found: ID does not exist" containerID="826f76e412cdc93a7fafbadc88cb90b8800f645bff0e7a2f0eaa2ea926dea50f"
Sep 17 19:10:00.822373 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:00.822316    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"826f76e412cdc93a7fafbadc88cb90b8800f645bff0e7a2f0eaa2ea926dea50f"} err="failed to get container status \"826f76e412cdc93a7fafbadc88cb90b8800f645bff0e7a2f0eaa2ea926dea50f\": rpc error: code = NotFound desc = could not find container \"826f76e412cdc93a7fafbadc88cb90b8800f645bff0e7a2f0eaa2ea926dea50f\": container with ID starting with 826f76e412cdc93a7fafbadc88cb90b8800f645bff0e7a2f0eaa2ea926dea50f not found: ID does not exist"
Sep 17 19:10:00.907020 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:00.906997    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fcbbae6-f4a3-420e-a433-e446903ebd85-tls-certs\") pod \"2fcbbae6-f4a3-420e-a433-e446903ebd85\" (UID: \"2fcbbae6-f4a3-420e-a433-e446903ebd85\") "
Sep 17 19:10:00.907115 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:00.907059    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5pr5\" (UniqueName: \"kubernetes.io/projected/2fcbbae6-f4a3-420e-a433-e446903ebd85-kube-api-access-f5pr5\") pod \"2fcbbae6-f4a3-420e-a433-e446903ebd85\" (UID: \"2fcbbae6-f4a3-420e-a433-e446903ebd85\") "
Sep 17 19:10:00.909124 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:00.909102    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fcbbae6-f4a3-420e-a433-e446903ebd85-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2fcbbae6-f4a3-420e-a433-e446903ebd85" (UID: "2fcbbae6-f4a3-420e-a433-e446903ebd85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:10:00.909213 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:00.909169    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fcbbae6-f4a3-420e-a433-e446903ebd85-kube-api-access-f5pr5" (OuterVolumeSpecName: "kube-api-access-f5pr5") pod "2fcbbae6-f4a3-420e-a433-e446903ebd85" (UID: "2fcbbae6-f4a3-420e-a433-e446903ebd85"). InnerVolumeSpecName "kube-api-access-f5pr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:10:01.008564 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:01.008510    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fcbbae6-f4a3-420e-a433-e446903ebd85-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:10:01.008564 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:01.008563    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5pr5\" (UniqueName: \"kubernetes.io/projected/2fcbbae6-f4a3-420e-a433-e446903ebd85-kube-api-access-f5pr5\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:10:01.133696 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:01.133665    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm"]
Sep 17 19:10:01.137123 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:01.137098    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58hxpgm"]
Sep 17 19:10:01.506584 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:01.506492    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fcbbae6-f4a3-420e-a433-e446903ebd85" path="/var/lib/kubelet/pods/2fcbbae6-f4a3-420e-a433-e446903ebd85/volumes"
Sep 17 19:10:05.475971 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:05.475934    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl" podUID="e247038a-662a-4efc-bbbb-2faa9f0976e2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:10:05.588611 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:05.588577    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:05.689369 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:05.689334    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:05.833642 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:05.833609    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:05.934988 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:05.934956    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:09.274508 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:09.274480    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:09.379883 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:09.379846    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl" podUID="e247038a-662a-4efc-bbbb-2faa9f0976e2" containerName="main" containerID="cri-o://7c6dccf0234e207d7a62383415c355a224ecb559e37cb0f1165cedd47edd394f" gracePeriod=45
Sep 17 19:10:12.001897 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:12.001868    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:15.476593 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:15.476554    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl" podUID="e247038a-662a-4efc-bbbb-2faa9f0976e2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:10:15.477038 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:15.476685    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl"
Sep 17 19:10:18.060507 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:18.060451    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:18.158803 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:18.158769    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:24.567090 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:24.567062    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:25.089761 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:25.089718    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp"]
Sep 17 19:10:25.090053 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:25.090039    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fcbbae6-f4a3-420e-a433-e446903ebd85" containerName="main"
Sep 17 19:10:25.090053 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:25.090053    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fcbbae6-f4a3-420e-a433-e446903ebd85" containerName="main"
Sep 17 19:10:25.090187 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:25.090111    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fcbbae6-f4a3-420e-a433-e446903ebd85" containerName="main"
Sep 17 19:10:25.092775 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:25.092753    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp"
Sep 17 19:10:25.097038 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:25.097015    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-2mk8s\""
Sep 17 19:10:25.097173 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:25.097040    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 17 19:10:25.097173 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:25.097015    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 17 19:10:25.097173 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:25.097103    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-4cc6b\""
Sep 17 19:10:25.097173 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:25.097019    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 17 19:10:25.107096 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:25.107076    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp"]
Sep 17 19:10:25.132631 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:25.132603    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:25.195926 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:25.195881    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp\" (UID: \"199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp"
Sep 17 19:10:25.196091 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:25.195930    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nvtc\" (UniqueName: \"kubernetes.io/projected/199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df-kube-api-access-4nvtc\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp\" (UID: \"199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp"
Sep 17 19:10:25.222958 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:25.222918    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:25.296775 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:25.296743    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp\" (UID: \"199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp"
Sep 17 19:10:25.296938 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:25.296782    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nvtc\" (UniqueName: \"kubernetes.io/projected/199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df-kube-api-access-4nvtc\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp\" (UID: \"199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp"
Sep 17 19:10:25.299367 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:25.299334    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp\" (UID: \"199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp"
Sep 17 19:10:25.305453 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:25.305419    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nvtc\" (UniqueName: \"kubernetes.io/projected/199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df-kube-api-access-4nvtc\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp\" (UID: \"199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp"
Sep 17 19:10:25.406129 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:25.406054    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-4cc6b\""
Sep 17 19:10:25.416241 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:25.416218    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-2mk8s\""
Sep 17 19:10:25.424279 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:25.424260    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp"
Sep 17 19:10:25.476333 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:25.476300    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl" podUID="e247038a-662a-4efc-bbbb-2faa9f0976e2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:10:25.695579 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:25.695475    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp"]
Sep 17 19:10:25.698548 ip-10-0-136-132 kubenswrapper[2588]: W0917 19:10:25.698504    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod199b2b0a_8b5e_4a40_8e0f_b7cc3b3521df.slice/crio-5e49fee2c99e2e81951a69f0e265d5b90cf8e599d278ee318ba7c50648b96453 WatchSource:0}: Error finding container 5e49fee2c99e2e81951a69f0e265d5b90cf8e599d278ee318ba7c50648b96453: Status 404 returned error can't find the container with id 5e49fee2c99e2e81951a69f0e265d5b90cf8e599d278ee318ba7c50648b96453
Sep 17 19:10:25.917554 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:25.917495    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp" event={"ID":"199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df","Type":"ContainerStarted","Data":"439d0f7468c9afd668cab4f1c56afdf72332230b2e69c79a2a56a914cd5868eb"}
Sep 17 19:10:25.917737 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:25.917582    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp"
Sep 17 19:10:25.917737 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:25.917606    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp" event={"ID":"199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df","Type":"ContainerStarted","Data":"5e49fee2c99e2e81951a69f0e265d5b90cf8e599d278ee318ba7c50648b96453"}
Sep 17 19:10:25.935152 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:25.935100    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp" podStartSLOduration=0.935084492 podStartE2EDuration="935.084492ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:10:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:10:25.933439035 +0000 UTC m=+2462.784444437" watchObservedRunningTime="2026-09-17 19:10:25.935084492 +0000 UTC m=+2462.786089891"
Sep 17 19:10:32.829548 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:32.829493    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:35.476613 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:35.476574    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl" podUID="e247038a-662a-4efc-bbbb-2faa9f0976e2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:10:39.624099 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:39.624073    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl"
Sep 17 19:10:39.702497 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:39.702405    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e247038a-662a-4efc-bbbb-2faa9f0976e2-tls-certs\") pod \"e247038a-662a-4efc-bbbb-2faa9f0976e2\" (UID: \"e247038a-662a-4efc-bbbb-2faa9f0976e2\") "
Sep 17 19:10:39.702497 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:39.702448    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhvxl\" (UniqueName: \"kubernetes.io/projected/e247038a-662a-4efc-bbbb-2faa9f0976e2-kube-api-access-dhvxl\") pod \"e247038a-662a-4efc-bbbb-2faa9f0976e2\" (UID: \"e247038a-662a-4efc-bbbb-2faa9f0976e2\") "
Sep 17 19:10:39.704794 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:39.704757    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e247038a-662a-4efc-bbbb-2faa9f0976e2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e247038a-662a-4efc-bbbb-2faa9f0976e2" (UID: "e247038a-662a-4efc-bbbb-2faa9f0976e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:10:39.704927 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:39.704837    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e247038a-662a-4efc-bbbb-2faa9f0976e2-kube-api-access-dhvxl" (OuterVolumeSpecName: "kube-api-access-dhvxl") pod "e247038a-662a-4efc-bbbb-2faa9f0976e2" (UID: "e247038a-662a-4efc-bbbb-2faa9f0976e2"). InnerVolumeSpecName "kube-api-access-dhvxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:10:39.803624 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:39.803594    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e247038a-662a-4efc-bbbb-2faa9f0976e2-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:10:39.803624 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:39.803622    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhvxl\" (UniqueName: \"kubernetes.io/projected/e247038a-662a-4efc-bbbb-2faa9f0976e2-kube-api-access-dhvxl\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:10:39.975086 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:39.975004    2588 generic.go:356] "Generic (PLEG): container finished" podID="e247038a-662a-4efc-bbbb-2faa9f0976e2" containerID="7c6dccf0234e207d7a62383415c355a224ecb559e37cb0f1165cedd47edd394f" exitCode=0
Sep 17 19:10:39.975086 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:39.975065    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl"
Sep 17 19:10:39.975247 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:39.975083    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl" event={"ID":"e247038a-662a-4efc-bbbb-2faa9f0976e2","Type":"ContainerDied","Data":"7c6dccf0234e207d7a62383415c355a224ecb559e37cb0f1165cedd47edd394f"}
Sep 17 19:10:39.975247 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:39.975157    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl" event={"ID":"e247038a-662a-4efc-bbbb-2faa9f0976e2","Type":"ContainerDied","Data":"a3b865dbf6cdd5ebbdc79d00fa1c2dbad5355be53a49ffc07bb2c38debbf584c"}
Sep 17 19:10:39.975247 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:39.975180    2588 scope.go:117] "RemoveContainer" containerID="7c6dccf0234e207d7a62383415c355a224ecb559e37cb0f1165cedd47edd394f"
Sep 17 19:10:39.984038 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:39.984023    2588 scope.go:117] "RemoveContainer" containerID="7c6dccf0234e207d7a62383415c355a224ecb559e37cb0f1165cedd47edd394f"
Sep 17 19:10:39.984276 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:10:39.984256    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c6dccf0234e207d7a62383415c355a224ecb559e37cb0f1165cedd47edd394f\": container with ID starting with 7c6dccf0234e207d7a62383415c355a224ecb559e37cb0f1165cedd47edd394f not found: ID does not exist" containerID="7c6dccf0234e207d7a62383415c355a224ecb559e37cb0f1165cedd47edd394f"
Sep 17 19:10:39.984349 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:39.984286    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c6dccf0234e207d7a62383415c355a224ecb559e37cb0f1165cedd47edd394f"} err="failed to get container status \"7c6dccf0234e207d7a62383415c355a224ecb559e37cb0f1165cedd47edd394f\": rpc error: code = NotFound desc = could not find container \"7c6dccf0234e207d7a62383415c355a224ecb559e37cb0f1165cedd47edd394f\": container with ID starting with 7c6dccf0234e207d7a62383415c355a224ecb559e37cb0f1165cedd47edd394f not found: ID does not exist"
Sep 17 19:10:39.996255 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:39.996225    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl"]
Sep 17 19:10:39.999980 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:39.999956    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche2c7zl"]
Sep 17 19:10:41.505905 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:41.505874    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e247038a-662a-4efc-bbbb-2faa9f0976e2" path="/var/lib/kubelet/pods/e247038a-662a-4efc-bbbb-2faa9f0976e2/volumes"
Sep 17 19:10:44.455886 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:44.455856    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:44.548816 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:44.548784    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:47.507841 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.507806    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"]
Sep 17 19:10:47.508207 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.508182    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e247038a-662a-4efc-bbbb-2faa9f0976e2" containerName="main"
Sep 17 19:10:47.508207 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.508198    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="e247038a-662a-4efc-bbbb-2faa9f0976e2" containerName="main"
Sep 17 19:10:47.508302 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.508290    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="e247038a-662a-4efc-bbbb-2faa9f0976e2" containerName="main"
Sep 17 19:10:47.512965 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.512944    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"
Sep 17 19:10:47.515374 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.515347    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 17 19:10:47.515495 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.515397    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:10:47.515495 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.515416    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-cqxf2\""
Sep 17 19:10:47.515495 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.515358    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 17 19:10:47.515821 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.515801    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 17 19:10:47.515928 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.515911    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 17 19:10:47.523144 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.523123    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"]
Sep 17 19:10:47.536497 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.536477    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:47.566967 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.566941    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc4hj\" (UniqueName: \"kubernetes.io/projected/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-kube-api-access-mc4hj\") pod \"stop-feature-test-kserve-7d8f57947b-w2cmb\" (UID: \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"
Sep 17 19:10:47.567070 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.566972    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-model-cache\") pod \"stop-feature-test-kserve-7d8f57947b-w2cmb\" (UID: \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"
Sep 17 19:10:47.567070 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.566999    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-kserve-provision-location\") pod \"stop-feature-test-kserve-7d8f57947b-w2cmb\" (UID: \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"
Sep 17 19:10:47.567070 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.567018    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-home\") pod \"stop-feature-test-kserve-7d8f57947b-w2cmb\" (UID: \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"
Sep 17 19:10:47.567070 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.567033    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-dshm\") pod \"stop-feature-test-kserve-7d8f57947b-w2cmb\" (UID: \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"
Sep 17 19:10:47.567207 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.567108    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-cabundle-cert\") pod \"stop-feature-test-kserve-7d8f57947b-w2cmb\" (UID: \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"
Sep 17 19:10:47.567207 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.567144    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-tls-certs\") pod \"stop-feature-test-kserve-7d8f57947b-w2cmb\" (UID: \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"
Sep 17 19:10:47.567207 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.567185    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-tmp-dir\") pod \"stop-feature-test-kserve-7d8f57947b-w2cmb\" (UID: \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"
Sep 17 19:10:47.668284 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.668251    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-tmp-dir\") pod \"stop-feature-test-kserve-7d8f57947b-w2cmb\" (UID: \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"
Sep 17 19:10:47.668284 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.668285    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc4hj\" (UniqueName: \"kubernetes.io/projected/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-kube-api-access-mc4hj\") pod \"stop-feature-test-kserve-7d8f57947b-w2cmb\" (UID: \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"
Sep 17 19:10:47.668489 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.668303    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-model-cache\") pod \"stop-feature-test-kserve-7d8f57947b-w2cmb\" (UID: \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"
Sep 17 19:10:47.668489 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.668334    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-kserve-provision-location\") pod \"stop-feature-test-kserve-7d8f57947b-w2cmb\" (UID: \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"
Sep 17 19:10:47.668489 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.668365    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-home\") pod \"stop-feature-test-kserve-7d8f57947b-w2cmb\" (UID: \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"
Sep 17 19:10:47.668489 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.668388    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-dshm\") pod \"stop-feature-test-kserve-7d8f57947b-w2cmb\" (UID: \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"
Sep 17 19:10:47.668489 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.668430    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-cabundle-cert\") pod \"stop-feature-test-kserve-7d8f57947b-w2cmb\" (UID: \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"
Sep 17 19:10:47.668489 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.668465    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-tls-certs\") pod \"stop-feature-test-kserve-7d8f57947b-w2cmb\" (UID: \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"
Sep 17 19:10:47.668828 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.668727    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-tmp-dir\") pod \"stop-feature-test-kserve-7d8f57947b-w2cmb\" (UID: \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"
Sep 17 19:10:47.668828 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.668764    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-model-cache\") pod \"stop-feature-test-kserve-7d8f57947b-w2cmb\" (UID: \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"
Sep 17 19:10:47.668945 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.668888    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-home\") pod \"stop-feature-test-kserve-7d8f57947b-w2cmb\" (UID: \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"
Sep 17 19:10:47.668999 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.668952    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-kserve-provision-location\") pod \"stop-feature-test-kserve-7d8f57947b-w2cmb\" (UID: \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"
Sep 17 19:10:47.669234 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.669214    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-cabundle-cert\") pod \"stop-feature-test-kserve-7d8f57947b-w2cmb\" (UID: \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"
Sep 17 19:10:47.670818 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.670796    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-dshm\") pod \"stop-feature-test-kserve-7d8f57947b-w2cmb\" (UID: \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"
Sep 17 19:10:47.671067 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.671048    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-tls-certs\") pod \"stop-feature-test-kserve-7d8f57947b-w2cmb\" (UID: \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"
Sep 17 19:10:47.676672 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.676649    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc4hj\" (UniqueName: \"kubernetes.io/projected/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-kube-api-access-mc4hj\") pod \"stop-feature-test-kserve-7d8f57947b-w2cmb\" (UID: \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"
Sep 17 19:10:47.712698 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.712674    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq"]
Sep 17 19:10:47.716947 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.716931    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq"
Sep 17 19:10:47.719243 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.719221    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-hktpq\""
Sep 17 19:10:47.728294 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.728274    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq"]
Sep 17 19:10:47.749138 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.749118    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:47.769362 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.769315    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17e5616e-4770-4b2d-98e5-ce5426e1e760-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq\" (UID: \"17e5616e-4770-4b2d-98e5-ce5426e1e760\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq"
Sep 17 19:10:47.769421 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.769361    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49ccs\" (UniqueName: \"kubernetes.io/projected/17e5616e-4770-4b2d-98e5-ce5426e1e760-kube-api-access-49ccs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq\" (UID: \"17e5616e-4770-4b2d-98e5-ce5426e1e760\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq"
Sep 17 19:10:47.824408 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.824382    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"
Sep 17 19:10:47.870282 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.869831    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49ccs\" (UniqueName: \"kubernetes.io/projected/17e5616e-4770-4b2d-98e5-ce5426e1e760-kube-api-access-49ccs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq\" (UID: \"17e5616e-4770-4b2d-98e5-ce5426e1e760\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq"
Sep 17 19:10:47.870282 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.870029    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17e5616e-4770-4b2d-98e5-ce5426e1e760-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq\" (UID: \"17e5616e-4770-4b2d-98e5-ce5426e1e760\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq"
Sep 17 19:10:47.873302 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.873251    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17e5616e-4770-4b2d-98e5-ce5426e1e760-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq\" (UID: \"17e5616e-4770-4b2d-98e5-ce5426e1e760\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq"
Sep 17 19:10:47.882584 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:47.882556    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49ccs\" (UniqueName: \"kubernetes.io/projected/17e5616e-4770-4b2d-98e5-ce5426e1e760-kube-api-access-49ccs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq\" (UID: \"17e5616e-4770-4b2d-98e5-ce5426e1e760\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq"
Sep 17 19:10:48.026550 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:48.026434    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq"
Sep 17 19:10:48.103364 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:48.103333    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"]
Sep 17 19:10:48.104568 ip-10-0-136-132 kubenswrapper[2588]: W0917 19:10:48.104539    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6b31f56_e7f8_4875_94ed_c72f40f4ed67.slice/crio-95e50cf023a033e7943834b05672be1f597ff6a0e1d655b52e6e3cf753a74e2a WatchSource:0}: Error finding container 95e50cf023a033e7943834b05672be1f597ff6a0e1d655b52e6e3cf753a74e2a: Status 404 returned error can't find the container with id 95e50cf023a033e7943834b05672be1f597ff6a0e1d655b52e6e3cf753a74e2a
Sep 17 19:10:48.303275 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:48.303232    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq"]
Sep 17 19:10:48.307792 ip-10-0-136-132 kubenswrapper[2588]: W0917 19:10:48.307760    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e5616e_4770_4b2d_98e5_ce5426e1e760.slice/crio-d2429b42bd48c9f8517ca245657947c4c8eb594657709c9b3387824e304e172a WatchSource:0}: Error finding container d2429b42bd48c9f8517ca245657947c4c8eb594657709c9b3387824e304e172a: Status 404 returned error can't find the container with id d2429b42bd48c9f8517ca245657947c4c8eb594657709c9b3387824e304e172a
Sep 17 19:10:49.018281 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:49.017357    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb" event={"ID":"e6b31f56-e7f8-4875-94ed-c72f40f4ed67","Type":"ContainerStarted","Data":"6365a60e096d7773807bddfe562baad35f5f94aecc14bb98896147a38e1fbe9e"}
Sep 17 19:10:49.018281 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:49.017424    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb" event={"ID":"e6b31f56-e7f8-4875-94ed-c72f40f4ed67","Type":"ContainerStarted","Data":"95e50cf023a033e7943834b05672be1f597ff6a0e1d655b52e6e3cf753a74e2a"}
Sep 17 19:10:49.020558 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:49.020505    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq" event={"ID":"17e5616e-4770-4b2d-98e5-ce5426e1e760","Type":"ContainerStarted","Data":"4c477067de9a564d972e3169cea77f694390288d7098dd692128438c1c46f27d"}
Sep 17 19:10:49.020791 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:49.020773    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq" event={"ID":"17e5616e-4770-4b2d-98e5-ce5426e1e760","Type":"ContainerStarted","Data":"d2429b42bd48c9f8517ca245657947c4c8eb594657709c9b3387824e304e172a"}
Sep 17 19:10:49.020998 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:49.020984    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq"
Sep 17 19:10:49.058936 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:49.058846    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq" podStartSLOduration=2.058831741 podStartE2EDuration="2.058831741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:10:49.057046669 +0000 UTC m=+2485.908052067" watchObservedRunningTime="2026-09-17 19:10:49.058831741 +0000 UTC m=+2485.909837139"
Sep 17 19:10:51.033095 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:51.033003    2588 generic.go:356] "Generic (PLEG): container finished" podID="e6b31f56-e7f8-4875-94ed-c72f40f4ed67" containerID="6365a60e096d7773807bddfe562baad35f5f94aecc14bb98896147a38e1fbe9e" exitCode=0
Sep 17 19:10:51.033095 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:51.033039    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb" event={"ID":"e6b31f56-e7f8-4875-94ed-c72f40f4ed67","Type":"ContainerDied","Data":"6365a60e096d7773807bddfe562baad35f5f94aecc14bb98896147a38e1fbe9e"}
Sep 17 19:10:51.075948 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:51.075898    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:52.039922 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:52.039888    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb" event={"ID":"e6b31f56-e7f8-4875-94ed-c72f40f4ed67","Type":"ContainerStarted","Data":"62e5f9f5fe7da9133dd3e5840351b48b2dd6e144d47c65291f2316a88a7cefe2"}
Sep 17 19:10:52.064611 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:52.064564    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb" podStartSLOduration=2.498145744 podStartE2EDuration="5.064545184s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.56639944s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:10:52.063879942 +0000 UTC m=+2488.914885337" watchObservedRunningTime="2026-09-17 19:10:52.064545184 +0000 UTC m=+2488.915550579"
Sep 17 19:10:52.825353 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:52.825311    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"
Sep 17 19:10:54.860064 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:54.860031    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:56.727793 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:56.727753    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:56.803910 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:56.803873    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:56.924218 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:56.924187    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp"
Sep 17 19:10:57.825577 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:57.825544    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"
Sep 17 19:10:57.827148 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:10:57.827120    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb" podUID="e6b31f56-e7f8-4875-94ed-c72f40f4ed67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 19:11:07.362861 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:11:07.362779    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:11:07.425000 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:11:07.424949    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:11:07.825699 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:11:07.825659    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb" podUID="e6b31f56-e7f8-4875-94ed-c72f40f4ed67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 19:11:17.825508 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:11:17.825463    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb" podUID="e6b31f56-e7f8-4875-94ed-c72f40f4ed67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 19:11:20.029874 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:11:20.029844    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq"
Sep 17 19:11:27.824852 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:11:27.824808    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb" podUID="e6b31f56-e7f8-4875-94ed-c72f40f4ed67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 19:11:37.825508 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:11:37.825465    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb" podUID="e6b31f56-e7f8-4875-94ed-c72f40f4ed67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 19:11:47.824822 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:11:47.824775    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb" podUID="e6b31f56-e7f8-4875-94ed-c72f40f4ed67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 19:11:57.825055 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:11:57.825010    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb" podUID="e6b31f56-e7f8-4875-94ed-c72f40f4ed67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 19:12:07.825342 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:12:07.825291    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb" podUID="e6b31f56-e7f8-4875-94ed-c72f40f4ed67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 19:12:17.825756 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:12:17.825707    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb" podUID="e6b31f56-e7f8-4875-94ed-c72f40f4ed67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 19:12:27.825001 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:12:27.824959    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb" podUID="e6b31f56-e7f8-4875-94ed-c72f40f4ed67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 19:12:29.604060 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:12:29.604019    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:12:29.615796 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:12:29.615766    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp"]
Sep 17 19:12:30.777128 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:12:30.777090    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:12:36.923450 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:12:36.923359    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp" podUID="199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:12:37.825765 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:12:37.825718    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb" podUID="e6b31f56-e7f8-4875-94ed-c72f40f4ed67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 19:12:43.694051 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:12:43.694011    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:12:43.762362 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:12:43.762327    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:12:44.635072 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:12:44.635020    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp" podUID="199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df" containerName="main" containerID="cri-o://439d0f7468c9afd668cab4f1c56afdf72332230b2e69c79a2a56a914cd5868eb" gracePeriod=45
Sep 17 19:12:46.924165 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:12:46.924128    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp" podUID="199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:12:47.835269 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:12:47.835228    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"
Sep 17 19:12:47.843068 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:12:47.843040    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"
Sep 17 19:12:49.280574 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:12:49.280519    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"]
Sep 17 19:12:49.314845 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:12:49.314811    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq"]
Sep 17 19:12:49.318410 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:12:49.318386    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:12:49.441264 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:12:49.441233    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:12:49.541585 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:12:49.541494    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:12:50.028864 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:12:50.028830    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq" podUID="17e5616e-4770-4b2d-98e5-ce5426e1e760" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:12:56.924065 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:12:56.924026    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp" podUID="199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:12:56.924454 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:12:56.924129    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp"
Sep 17 19:13:00.028710 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:00.028671    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq" podUID="17e5616e-4770-4b2d-98e5-ce5426e1e760" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:13:04.335893 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:04.335845    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq" podUID="17e5616e-4770-4b2d-98e5-ce5426e1e760" containerName="main" containerID="cri-o://4c477067de9a564d972e3169cea77f694390288d7098dd692128438c1c46f27d" gracePeriod=45
Sep 17 19:13:04.595720 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:04.595593    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb" podUID="e6b31f56-e7f8-4875-94ed-c72f40f4ed67" containerName="main" containerID="cri-o://62e5f9f5fe7da9133dd3e5840351b48b2dd6e144d47c65291f2316a88a7cefe2" gracePeriod=45
Sep 17 19:13:04.836197 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:04.836151    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb" podUID="e6b31f56-e7f8-4875-94ed-c72f40f4ed67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 19:13:05.247559 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.247517    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"
Sep 17 19:13:05.351400 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.351299    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-model-cache\") pod \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\" (UID: \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\") "
Sep 17 19:13:05.351400 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.351362    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-kserve-provision-location\") pod \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\" (UID: \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\") "
Sep 17 19:13:05.351400 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.351387    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-tls-certs\") pod \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\" (UID: \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\") "
Sep 17 19:13:05.352027 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.351416    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-tmp-dir\") pod \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\" (UID: \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\") "
Sep 17 19:13:05.352027 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.351449    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-cabundle-cert\") pod \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\" (UID: \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\") "
Sep 17 19:13:05.352027 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.351472    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mc4hj\" (UniqueName: \"kubernetes.io/projected/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-kube-api-access-mc4hj\") pod \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\" (UID: \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\") "
Sep 17 19:13:05.352027 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.351558    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-home\") pod \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\" (UID: \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\") "
Sep 17 19:13:05.352027 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.351623    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-dshm\") pod \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\" (UID: \"e6b31f56-e7f8-4875-94ed-c72f40f4ed67\") "
Sep 17 19:13:05.352027 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.351691    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-model-cache" (OuterVolumeSpecName: "model-cache") pod "e6b31f56-e7f8-4875-94ed-c72f40f4ed67" (UID: "e6b31f56-e7f8-4875-94ed-c72f40f4ed67"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:13:05.352027 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.351881    2588 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-model-cache\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:13:05.352451 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.352175    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e6b31f56-e7f8-4875-94ed-c72f40f4ed67" (UID: "e6b31f56-e7f8-4875-94ed-c72f40f4ed67"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:13:05.352451 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.352429    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-home" (OuterVolumeSpecName: "home") pod "e6b31f56-e7f8-4875-94ed-c72f40f4ed67" (UID: "e6b31f56-e7f8-4875-94ed-c72f40f4ed67"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:13:05.354082 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.354050    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-dshm" (OuterVolumeSpecName: "dshm") pod "e6b31f56-e7f8-4875-94ed-c72f40f4ed67" (UID: "e6b31f56-e7f8-4875-94ed-c72f40f4ed67"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:13:05.354250 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.354227    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e6b31f56-e7f8-4875-94ed-c72f40f4ed67" (UID: "e6b31f56-e7f8-4875-94ed-c72f40f4ed67"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:13:05.354371 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.354347    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-kube-api-access-mc4hj" (OuterVolumeSpecName: "kube-api-access-mc4hj") pod "e6b31f56-e7f8-4875-94ed-c72f40f4ed67" (UID: "e6b31f56-e7f8-4875-94ed-c72f40f4ed67"). InnerVolumeSpecName "kube-api-access-mc4hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:13:05.363333 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.363304    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e6b31f56-e7f8-4875-94ed-c72f40f4ed67" (UID: "e6b31f56-e7f8-4875-94ed-c72f40f4ed67"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:13:05.374241 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.374214    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e6b31f56-e7f8-4875-94ed-c72f40f4ed67" (UID: "e6b31f56-e7f8-4875-94ed-c72f40f4ed67"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:13:05.452862 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.452833    2588 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-dshm\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:13:05.452862 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.452860    2588 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-kserve-provision-location\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:13:05.453026 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.452870    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:13:05.453026 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.452879    2588 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-tmp-dir\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:13:05.453026 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.452886    2588 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-cabundle-cert\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:13:05.453026 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.452894    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc4hj\" (UniqueName: \"kubernetes.io/projected/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-kube-api-access-mc4hj\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:13:05.453026 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.452901    2588 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6b31f56-e7f8-4875-94ed-c72f40f4ed67-home\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:13:05.637620 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.637510    2588 generic.go:356] "Generic (PLEG): container finished" podID="e6b31f56-e7f8-4875-94ed-c72f40f4ed67" containerID="62e5f9f5fe7da9133dd3e5840351b48b2dd6e144d47c65291f2316a88a7cefe2" exitCode=0
Sep 17 19:13:05.637620 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.637558    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb" event={"ID":"e6b31f56-e7f8-4875-94ed-c72f40f4ed67","Type":"ContainerDied","Data":"62e5f9f5fe7da9133dd3e5840351b48b2dd6e144d47c65291f2316a88a7cefe2"}
Sep 17 19:13:05.637620 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.637581    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"
Sep 17 19:13:05.637620 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.637614    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb" event={"ID":"e6b31f56-e7f8-4875-94ed-c72f40f4ed67","Type":"ContainerDied","Data":"95e50cf023a033e7943834b05672be1f597ff6a0e1d655b52e6e3cf753a74e2a"}
Sep 17 19:13:05.637963 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.637636    2588 scope.go:117] "RemoveContainer" containerID="62e5f9f5fe7da9133dd3e5840351b48b2dd6e144d47c65291f2316a88a7cefe2"
Sep 17 19:13:05.646042 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.646022    2588 scope.go:117] "RemoveContainer" containerID="6365a60e096d7773807bddfe562baad35f5f94aecc14bb98896147a38e1fbe9e"
Sep 17 19:13:05.653230 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.653211    2588 scope.go:117] "RemoveContainer" containerID="62e5f9f5fe7da9133dd3e5840351b48b2dd6e144d47c65291f2316a88a7cefe2"
Sep 17 19:13:05.653478 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:13:05.653460    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62e5f9f5fe7da9133dd3e5840351b48b2dd6e144d47c65291f2316a88a7cefe2\": container with ID starting with 62e5f9f5fe7da9133dd3e5840351b48b2dd6e144d47c65291f2316a88a7cefe2 not found: ID does not exist" containerID="62e5f9f5fe7da9133dd3e5840351b48b2dd6e144d47c65291f2316a88a7cefe2"
Sep 17 19:13:05.653542 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.653486    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e5f9f5fe7da9133dd3e5840351b48b2dd6e144d47c65291f2316a88a7cefe2"} err="failed to get container status \"62e5f9f5fe7da9133dd3e5840351b48b2dd6e144d47c65291f2316a88a7cefe2\": rpc error: code = NotFound desc = could not find container \"62e5f9f5fe7da9133dd3e5840351b48b2dd6e144d47c65291f2316a88a7cefe2\": container with ID starting with 62e5f9f5fe7da9133dd3e5840351b48b2dd6e144d47c65291f2316a88a7cefe2 not found: ID does not exist"
Sep 17 19:13:05.653542 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.653501    2588 scope.go:117] "RemoveContainer" containerID="6365a60e096d7773807bddfe562baad35f5f94aecc14bb98896147a38e1fbe9e"
Sep 17 19:13:05.653777 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:13:05.653756    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6365a60e096d7773807bddfe562baad35f5f94aecc14bb98896147a38e1fbe9e\": container with ID starting with 6365a60e096d7773807bddfe562baad35f5f94aecc14bb98896147a38e1fbe9e not found: ID does not exist" containerID="6365a60e096d7773807bddfe562baad35f5f94aecc14bb98896147a38e1fbe9e"
Sep 17 19:13:05.653849 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.653775    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6365a60e096d7773807bddfe562baad35f5f94aecc14bb98896147a38e1fbe9e"} err="failed to get container status \"6365a60e096d7773807bddfe562baad35f5f94aecc14bb98896147a38e1fbe9e\": rpc error: code = NotFound desc = could not find container \"6365a60e096d7773807bddfe562baad35f5f94aecc14bb98896147a38e1fbe9e\": container with ID starting with 6365a60e096d7773807bddfe562baad35f5f94aecc14bb98896147a38e1fbe9e not found: ID does not exist"
Sep 17 19:13:05.656220 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.656200    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"]
Sep 17 19:13:05.660909 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:05.660889    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-w2cmb"]
Sep 17 19:13:06.441275 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:06.441240    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:06.707949 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:06.707864    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:06.923760 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:06.923722    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp" podUID="199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:13:07.506965 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:07.506929    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6b31f56-e7f8-4875-94ed-c72f40f4ed67" path="/var/lib/kubelet/pods/e6b31f56-e7f8-4875-94ed-c72f40f4ed67/volumes"
Sep 17 19:13:10.029517 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:10.029480    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq" podUID="17e5616e-4770-4b2d-98e5-ce5426e1e760" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:13:10.030011 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:10.029598    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq"
Sep 17 19:13:11.919481 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:11.919438    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:14.225910 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:14.225874    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:14.878145 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:14.878116    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp"
Sep 17 19:13:14.919830 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:14.919750    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df-tls-certs\") pod \"199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df\" (UID: \"199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df\") "
Sep 17 19:13:14.919830 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:14.919795    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nvtc\" (UniqueName: \"kubernetes.io/projected/199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df-kube-api-access-4nvtc\") pod \"199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df\" (UID: \"199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df\") "
Sep 17 19:13:14.921996 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:14.921965    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df" (UID: "199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:13:14.922095 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:14.922021    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df-kube-api-access-4nvtc" (OuterVolumeSpecName: "kube-api-access-4nvtc") pod "199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df" (UID: "199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df"). InnerVolumeSpecName "kube-api-access-4nvtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:13:15.020299 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:15.020266    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:13:15.020299 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:15.020292    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nvtc\" (UniqueName: \"kubernetes.io/projected/199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df-kube-api-access-4nvtc\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:13:15.680722 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:15.680690    2588 generic.go:356] "Generic (PLEG): container finished" podID="199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df" containerID="439d0f7468c9afd668cab4f1c56afdf72332230b2e69c79a2a56a914cd5868eb" exitCode=0
Sep 17 19:13:15.681208 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:15.680734    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp"
Sep 17 19:13:15.681208 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:15.680740    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp" event={"ID":"199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df","Type":"ContainerDied","Data":"439d0f7468c9afd668cab4f1c56afdf72332230b2e69c79a2a56a914cd5868eb"}
Sep 17 19:13:15.681208 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:15.680793    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp" event={"ID":"199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df","Type":"ContainerDied","Data":"5e49fee2c99e2e81951a69f0e265d5b90cf8e599d278ee318ba7c50648b96453"}
Sep 17 19:13:15.681208 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:15.680825    2588 scope.go:117] "RemoveContainer" containerID="439d0f7468c9afd668cab4f1c56afdf72332230b2e69c79a2a56a914cd5868eb"
Sep 17 19:13:15.688905 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:15.688887    2588 scope.go:117] "RemoveContainer" containerID="439d0f7468c9afd668cab4f1c56afdf72332230b2e69c79a2a56a914cd5868eb"
Sep 17 19:13:15.689178 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:13:15.689157    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"439d0f7468c9afd668cab4f1c56afdf72332230b2e69c79a2a56a914cd5868eb\": container with ID starting with 439d0f7468c9afd668cab4f1c56afdf72332230b2e69c79a2a56a914cd5868eb not found: ID does not exist" containerID="439d0f7468c9afd668cab4f1c56afdf72332230b2e69c79a2a56a914cd5868eb"
Sep 17 19:13:15.689237 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:15.689190    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"439d0f7468c9afd668cab4f1c56afdf72332230b2e69c79a2a56a914cd5868eb"} err="failed to get container status \"439d0f7468c9afd668cab4f1c56afdf72332230b2e69c79a2a56a914cd5868eb\": rpc error: code = NotFound desc = could not find container \"439d0f7468c9afd668cab4f1c56afdf72332230b2e69c79a2a56a914cd5868eb\": container with ID starting with 439d0f7468c9afd668cab4f1c56afdf72332230b2e69c79a2a56a914cd5868eb not found: ID does not exist"
Sep 17 19:13:15.695206 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:15.695174    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp"]
Sep 17 19:13:15.699643 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:15.699623    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-zz4fp"]
Sep 17 19:13:17.506420 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:17.506386    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df" path="/var/lib/kubelet/pods/199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df/volumes"
Sep 17 19:13:19.700488 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:19.700456    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:19.795895 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:19.795863    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:20.028830 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:20.028749    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq" podUID="17e5616e-4770-4b2d-98e5-ce5426e1e760" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:13:24.739171 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:24.739141    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:29.198502 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.198459    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"]
Sep 17 19:13:29.200904 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.198757    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6b31f56-e7f8-4875-94ed-c72f40f4ed67" containerName="storage-initializer"
Sep 17 19:13:29.200904 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.198770    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6b31f56-e7f8-4875-94ed-c72f40f4ed67" containerName="storage-initializer"
Sep 17 19:13:29.200904 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.198789    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df" containerName="main"
Sep 17 19:13:29.200904 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.198794    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df" containerName="main"
Sep 17 19:13:29.200904 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.198800    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6b31f56-e7f8-4875-94ed-c72f40f4ed67" containerName="main"
Sep 17 19:13:29.200904 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.198805    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6b31f56-e7f8-4875-94ed-c72f40f4ed67" containerName="main"
Sep 17 19:13:29.200904 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.198852    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="199b2b0a-8b5e-4a40-8e0f-b7cc3b3521df" containerName="main"
Sep 17 19:13:29.200904 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.198859    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6b31f56-e7f8-4875-94ed-c72f40f4ed67" containerName="main"
Sep 17 19:13:29.201958 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.201934    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:13:29.204683 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.204657    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 17 19:13:29.204805 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.204706    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 17 19:13:29.204805 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.204671    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:13:29.205073 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.205059    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-5r7hp\""
Sep 17 19:13:29.205226 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.205206    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 17 19:13:29.205562 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.205544    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-qsj2m\""
Sep 17 19:13:29.205649 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.205596    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 17 19:13:29.222088 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.222049    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ad69a7e-adf0-4c53-912b-8f331d91eb43-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc\" (UID: \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:13:29.222196 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.222095    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ad69a7e-adf0-4c53-912b-8f331d91eb43-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc\" (UID: \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:13:29.222196 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.222120    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ad69a7e-adf0-4c53-912b-8f331d91eb43-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc\" (UID: \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:13:29.222196 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.222179    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwvnm\" (UniqueName: \"kubernetes.io/projected/7ad69a7e-adf0-4c53-912b-8f331d91eb43-kube-api-access-jwvnm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc\" (UID: \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:13:29.222307 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.222204    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ad69a7e-adf0-4c53-912b-8f331d91eb43-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc\" (UID: \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:13:29.222307 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.222273    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ad69a7e-adf0-4c53-912b-8f331d91eb43-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc\" (UID: \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:13:29.222370 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.222320    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ad69a7e-adf0-4c53-912b-8f331d91eb43-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc\" (UID: \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:13:29.222370 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.222346    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ad69a7e-adf0-4c53-912b-8f331d91eb43-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc\" (UID: \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:13:29.230514 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.230488    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"]
Sep 17 19:13:29.263973 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.263942    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:29.323230 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.323192    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ad69a7e-adf0-4c53-912b-8f331d91eb43-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc\" (UID: \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:13:29.323230 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.323233    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ad69a7e-adf0-4c53-912b-8f331d91eb43-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc\" (UID: \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:13:29.323472 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.323258    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ad69a7e-adf0-4c53-912b-8f331d91eb43-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc\" (UID: \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:13:29.323472 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.323306    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwvnm\" (UniqueName: \"kubernetes.io/projected/7ad69a7e-adf0-4c53-912b-8f331d91eb43-kube-api-access-jwvnm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc\" (UID: \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:13:29.323472 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.323338    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ad69a7e-adf0-4c53-912b-8f331d91eb43-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc\" (UID: \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:13:29.323472 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.323378    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ad69a7e-adf0-4c53-912b-8f331d91eb43-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc\" (UID: \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:13:29.323472 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.323413    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ad69a7e-adf0-4c53-912b-8f331d91eb43-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc\" (UID: \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:13:29.323472 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.323446    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ad69a7e-adf0-4c53-912b-8f331d91eb43-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc\" (UID: \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:13:29.323856 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.323676    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ad69a7e-adf0-4c53-912b-8f331d91eb43-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc\" (UID: \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:13:29.323917 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.323862    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ad69a7e-adf0-4c53-912b-8f331d91eb43-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc\" (UID: \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:13:29.323975 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.323917    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ad69a7e-adf0-4c53-912b-8f331d91eb43-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc\" (UID: \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:13:29.324109 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.324090    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ad69a7e-adf0-4c53-912b-8f331d91eb43-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc\" (UID: \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:13:29.324167 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.324147    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ad69a7e-adf0-4c53-912b-8f331d91eb43-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc\" (UID: \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:13:29.325792 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.325771    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ad69a7e-adf0-4c53-912b-8f331d91eb43-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc\" (UID: \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:13:29.326078 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.326062    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ad69a7e-adf0-4c53-912b-8f331d91eb43-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc\" (UID: \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:13:29.332359 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.332337    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwvnm\" (UniqueName: \"kubernetes.io/projected/7ad69a7e-adf0-4c53-912b-8f331d91eb43-kube-api-access-jwvnm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc\" (UID: \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:13:29.513582 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.513434    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:13:29.516588 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.516559    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7"]
Sep 17 19:13:29.520091 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.520070    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7"
Sep 17 19:13:29.522419 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.522397    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-tmk76\""
Sep 17 19:13:29.536660 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.536636    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7"]
Sep 17 19:13:29.564939 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.564904    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:29.626859 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.626826    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x59ws\" (UniqueName: \"kubernetes.io/projected/43108c2c-81b9-4a68-af16-35c0648a413e-kube-api-access-x59ws\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7\" (UID: \"43108c2c-81b9-4a68-af16-35c0648a413e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7"
Sep 17 19:13:29.627037 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.626901    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43108c2c-81b9-4a68-af16-35c0648a413e-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7\" (UID: \"43108c2c-81b9-4a68-af16-35c0648a413e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7"
Sep 17 19:13:29.728221 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.728184    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43108c2c-81b9-4a68-af16-35c0648a413e-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7\" (UID: \"43108c2c-81b9-4a68-af16-35c0648a413e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7"
Sep 17 19:13:29.728397 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.728251    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x59ws\" (UniqueName: \"kubernetes.io/projected/43108c2c-81b9-4a68-af16-35c0648a413e-kube-api-access-x59ws\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7\" (UID: \"43108c2c-81b9-4a68-af16-35c0648a413e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7"
Sep 17 19:13:29.730718 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.730694    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43108c2c-81b9-4a68-af16-35c0648a413e-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7\" (UID: \"43108c2c-81b9-4a68-af16-35c0648a413e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7"
Sep 17 19:13:29.739259 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.739238    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x59ws\" (UniqueName: \"kubernetes.io/projected/43108c2c-81b9-4a68-af16-35c0648a413e-kube-api-access-x59ws\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7\" (UID: \"43108c2c-81b9-4a68-af16-35c0648a413e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7"
Sep 17 19:13:29.800202 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.800169    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"]
Sep 17 19:13:29.803186 ip-10-0-136-132 kubenswrapper[2588]: W0917 19:13:29.803158    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad69a7e_adf0_4c53_912b_8f331d91eb43.slice/crio-d547e64342a3e1a22b35dfe37d05c92ba3079bba6e41a3ae2c1f17b4a29000b7 WatchSource:0}: Error finding container d547e64342a3e1a22b35dfe37d05c92ba3079bba6e41a3ae2c1f17b4a29000b7: Status 404 returned error can't find the container with id d547e64342a3e1a22b35dfe37d05c92ba3079bba6e41a3ae2c1f17b4a29000b7
Sep 17 19:13:29.805249 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.805233    2588 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:13:29.849621 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:29.849595    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7"
Sep 17 19:13:30.029412 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:30.029374    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq" podUID="17e5616e-4770-4b2d-98e5-ce5426e1e760" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:13:30.124265 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:30.124226    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7"]
Sep 17 19:13:30.126830 ip-10-0-136-132 kubenswrapper[2588]: W0917 19:13:30.126798    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43108c2c_81b9_4a68_af16_35c0648a413e.slice/crio-6de8710114acb98b49e6bcb3060f1b6468c4fa65d9843598aef54cebbbe50124 WatchSource:0}: Error finding container 6de8710114acb98b49e6bcb3060f1b6468c4fa65d9843598aef54cebbbe50124: Status 404 returned error can't find the container with id 6de8710114acb98b49e6bcb3060f1b6468c4fa65d9843598aef54cebbbe50124
Sep 17 19:13:30.744747 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:30.744698    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7" event={"ID":"43108c2c-81b9-4a68-af16-35c0648a413e","Type":"ContainerStarted","Data":"5235dd24667f83fea2f0d820518770bd0a81bcb3560c97d7b3c9691c77746e13"}
Sep 17 19:13:30.745224 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:30.744816    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7"
Sep 17 19:13:30.745224 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:30.744851    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7" event={"ID":"43108c2c-81b9-4a68-af16-35c0648a413e","Type":"ContainerStarted","Data":"6de8710114acb98b49e6bcb3060f1b6468c4fa65d9843598aef54cebbbe50124"}
Sep 17 19:13:30.749664 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:30.749603    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc" event={"ID":"7ad69a7e-adf0-4c53-912b-8f331d91eb43","Type":"ContainerStarted","Data":"d547e64342a3e1a22b35dfe37d05c92ba3079bba6e41a3ae2c1f17b4a29000b7"}
Sep 17 19:13:30.763250 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:30.763202    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7" podStartSLOduration=1.763185992 podStartE2EDuration="1.763185992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:13:30.762257852 +0000 UTC m=+2647.613263248" watchObservedRunningTime="2026-09-17 19:13:30.763185992 +0000 UTC m=+2647.614191392"
Sep 17 19:13:31.755999 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:31.755961    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc" event={"ID":"7ad69a7e-adf0-4c53-912b-8f331d91eb43","Type":"ContainerStarted","Data":"6caedbde7ee4aac325a6f7df88615c7eebc4ba44e776a3b8c2f4f98ceb1e0d7d"}
Sep 17 19:13:31.756434 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:31.756102    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:13:32.761866 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:32.761830    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc" event={"ID":"7ad69a7e-adf0-4c53-912b-8f331d91eb43","Type":"ContainerStarted","Data":"3da990122a44913375fe271cefae27305b8f477851273a3915903f146378e30e"}
Sep 17 19:13:34.578562 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:34.578516    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq"
Sep 17 19:13:34.671441 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:34.671353    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17e5616e-4770-4b2d-98e5-ce5426e1e760-tls-certs\") pod \"17e5616e-4770-4b2d-98e5-ce5426e1e760\" (UID: \"17e5616e-4770-4b2d-98e5-ce5426e1e760\") "
Sep 17 19:13:34.671441 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:34.671419    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49ccs\" (UniqueName: \"kubernetes.io/projected/17e5616e-4770-4b2d-98e5-ce5426e1e760-kube-api-access-49ccs\") pod \"17e5616e-4770-4b2d-98e5-ce5426e1e760\" (UID: \"17e5616e-4770-4b2d-98e5-ce5426e1e760\") "
Sep 17 19:13:34.673554 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:34.673517    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e5616e-4770-4b2d-98e5-ce5426e1e760-kube-api-access-49ccs" (OuterVolumeSpecName: "kube-api-access-49ccs") pod "17e5616e-4770-4b2d-98e5-ce5426e1e760" (UID: "17e5616e-4770-4b2d-98e5-ce5426e1e760"). InnerVolumeSpecName "kube-api-access-49ccs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:13:34.673666 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:34.673584    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17e5616e-4770-4b2d-98e5-ce5426e1e760-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "17e5616e-4770-4b2d-98e5-ce5426e1e760" (UID: "17e5616e-4770-4b2d-98e5-ce5426e1e760"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:13:34.771952 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:34.771921    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49ccs\" (UniqueName: \"kubernetes.io/projected/17e5616e-4770-4b2d-98e5-ce5426e1e760-kube-api-access-49ccs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:13:34.771952 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:34.771950    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17e5616e-4770-4b2d-98e5-ce5426e1e760-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:13:34.772157 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:34.772008    2588 generic.go:356] "Generic (PLEG): container finished" podID="7ad69a7e-adf0-4c53-912b-8f331d91eb43" containerID="3da990122a44913375fe271cefae27305b8f477851273a3915903f146378e30e" exitCode=0
Sep 17 19:13:34.772157 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:34.772078    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc" event={"ID":"7ad69a7e-adf0-4c53-912b-8f331d91eb43","Type":"ContainerDied","Data":"3da990122a44913375fe271cefae27305b8f477851273a3915903f146378e30e"}
Sep 17 19:13:34.773452 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:34.773426    2588 generic.go:356] "Generic (PLEG): container finished" podID="17e5616e-4770-4b2d-98e5-ce5426e1e760" containerID="4c477067de9a564d972e3169cea77f694390288d7098dd692128438c1c46f27d" exitCode=0
Sep 17 19:13:34.773595 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:34.773502    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq" event={"ID":"17e5616e-4770-4b2d-98e5-ce5426e1e760","Type":"ContainerDied","Data":"4c477067de9a564d972e3169cea77f694390288d7098dd692128438c1c46f27d"}
Sep 17 19:13:34.773595 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:34.773519    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq"
Sep 17 19:13:34.773595 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:34.773582    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq" event={"ID":"17e5616e-4770-4b2d-98e5-ce5426e1e760","Type":"ContainerDied","Data":"d2429b42bd48c9f8517ca245657947c4c8eb594657709c9b3387824e304e172a"}
Sep 17 19:13:34.773775 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:34.773615    2588 scope.go:117] "RemoveContainer" containerID="4c477067de9a564d972e3169cea77f694390288d7098dd692128438c1c46f27d"
Sep 17 19:13:34.784726 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:34.784710    2588 scope.go:117] "RemoveContainer" containerID="4c477067de9a564d972e3169cea77f694390288d7098dd692128438c1c46f27d"
Sep 17 19:13:34.787514 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:13:34.787490    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c477067de9a564d972e3169cea77f694390288d7098dd692128438c1c46f27d\": container with ID starting with 4c477067de9a564d972e3169cea77f694390288d7098dd692128438c1c46f27d not found: ID does not exist" containerID="4c477067de9a564d972e3169cea77f694390288d7098dd692128438c1c46f27d"
Sep 17 19:13:34.787623 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:34.787542    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c477067de9a564d972e3169cea77f694390288d7098dd692128438c1c46f27d"} err="failed to get container status \"4c477067de9a564d972e3169cea77f694390288d7098dd692128438c1c46f27d\": rpc error: code = NotFound desc = could not find container \"4c477067de9a564d972e3169cea77f694390288d7098dd692128438c1c46f27d\": container with ID starting with 4c477067de9a564d972e3169cea77f694390288d7098dd692128438c1c46f27d not found: ID does not exist"
Sep 17 19:13:34.809165 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:34.809135    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq"]
Sep 17 19:13:34.818840 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:34.818817    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-f5gjq"]
Sep 17 19:13:35.508918 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:35.508882    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17e5616e-4770-4b2d-98e5-ce5426e1e760" path="/var/lib/kubelet/pods/17e5616e-4770-4b2d-98e5-ce5426e1e760/volumes"
Sep 17 19:13:35.780673 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:35.780576    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc" event={"ID":"7ad69a7e-adf0-4c53-912b-8f331d91eb43","Type":"ContainerStarted","Data":"c543e06f10389665e55d5fac4176c8f9da2b630df33fbb377736f8d453ce27d8"}
Sep 17 19:13:35.809421 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:35.809349    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc" podStartSLOduration=2.963735316 podStartE2EDuration="6.809328929s" totalImagesPullingTime="1.771360325s" totalInitContainerRuntime="2.074233288s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:13:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:13:35.808701046 +0000 UTC m=+2652.659706448" watchObservedRunningTime="2026-09-17 19:13:35.809328929 +0000 UTC m=+2652.660334329"
Sep 17 19:13:39.513740 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:39.513679    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:13:39.514264 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:39.513763    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:13:39.515066 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:39.515034    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc" podUID="7ad69a7e-adf0-4c53-912b-8f331d91eb43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Sep 17 19:13:44.881721 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:44.881682    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:47.012029 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:47.011926    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:47.736826 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:47.736790    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:49.514750 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:49.514699    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc" podUID="7ad69a7e-adf0-4c53-912b-8f331d91eb43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Sep 17 19:13:49.533365 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:49.533342    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:13:53.356519 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:53.356469    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:56.614857 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:56.614818    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:56.670576 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:56.670503    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:59.514486 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:13:59.514443    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc" podUID="7ad69a7e-adf0-4c53-912b-8f331d91eb43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Sep 17 19:14:01.758377 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:14:01.758350    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7"
Sep 17 19:14:09.514809 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:14:09.514711    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc" podUID="7ad69a7e-adf0-4c53-912b-8f331d91eb43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Sep 17 19:14:19.514647 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:14:19.514601    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc" podUID="7ad69a7e-adf0-4c53-912b-8f331d91eb43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Sep 17 19:14:25.653710 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:14:25.653677    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xlklb_07ee31e0-43bb-4940-837d-969bd840492b/console-operator/1.log"
Sep 17 19:14:25.657910 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:14:25.657888    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xlklb_07ee31e0-43bb-4940-837d-969bd840492b/console-operator/1.log"
Sep 17 19:14:25.659358 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:14:25.659334    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovn-acl-logging/0.log"
Sep 17 19:14:25.663449 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:14:25.663430    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovn-acl-logging/0.log"
Sep 17 19:14:28.399202 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:14:28.399169    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:14:29.514716 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:14:29.514674    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc" podUID="7ad69a7e-adf0-4c53-912b-8f331d91eb43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Sep 17 19:14:33.394420 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:14:33.394387    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:14:33.477835 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:14:33.477803    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:14:39.514079 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:14:39.514022    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc" podUID="7ad69a7e-adf0-4c53-912b-8f331d91eb43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Sep 17 19:14:49.513997 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:14:49.513953    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc" podUID="7ad69a7e-adf0-4c53-912b-8f331d91eb43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Sep 17 19:14:59.514108 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:14:59.514055    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc" podUID="7ad69a7e-adf0-4c53-912b-8f331d91eb43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Sep 17 19:15:09.514456 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:15:09.514401    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc" podUID="7ad69a7e-adf0-4c53-912b-8f331d91eb43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Sep 17 19:15:18.445921 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:15:18.445889    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:15:18.512623 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:15:18.512589    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:15:19.514286 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:15:19.514238    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc" podUID="7ad69a7e-adf0-4c53-912b-8f331d91eb43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Sep 17 19:15:29.529332 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:15:29.529303    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:15:29.549482 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:15:29.549460    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:15:49.904397 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:15:49.904360    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:15:50.006318 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:15:50.006283    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:15:58.888449 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:15:58.888415    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:15:58.900049 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:15:58.900017    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7"]
Sep 17 19:15:58.910948 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:15:58.910914    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"]
Sep 17 19:15:58.988130 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:15:58.988093    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:16:01.758476 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:01.758443    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7" podUID="43108c2c-81b9-4a68-af16-35c0648a413e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:16:08.517320 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:08.517275    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:16:08.631964 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:08.631933    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:16:08.718838 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:08.718811    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:16:11.757799 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:11.757765    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7" podUID="43108c2c-81b9-4a68-af16-35c0648a413e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:16:13.919417 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:13.919378    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7" podUID="43108c2c-81b9-4a68-af16-35c0648a413e" containerName="main" containerID="cri-o://5235dd24667f83fea2f0d820518770bd0a81bcb3560c97d7b3c9691c77746e13" gracePeriod=45
Sep 17 19:16:13.933970 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:13.933930    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc" podUID="7ad69a7e-adf0-4c53-912b-8f331d91eb43" containerName="main" containerID="cri-o://c543e06f10389665e55d5fac4176c8f9da2b630df33fbb377736f8d453ce27d8" gracePeriod=45
Sep 17 19:16:14.551390 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:14.551353    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc" podUID="7ad69a7e-adf0-4c53-912b-8f331d91eb43" containerName="llm-d-routing-sidecar" containerID="cri-o://6caedbde7ee4aac325a6f7df88615c7eebc4ba44e776a3b8c2f4f98ceb1e0d7d" gracePeriod=45
Sep 17 19:16:14.739821 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:14.739783    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:16:14.796170 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:14.796147    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:16:14.909009 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:14.908978    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ad69a7e-adf0-4c53-912b-8f331d91eb43-dshm\") pod \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\" (UID: \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\") "
Sep 17 19:16:14.909009 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:14.909012    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ad69a7e-adf0-4c53-912b-8f331d91eb43-tls-certs\") pod \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\" (UID: \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\") "
Sep 17 19:16:14.909261 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:14.909041    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ad69a7e-adf0-4c53-912b-8f331d91eb43-cabundle-cert\") pod \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\" (UID: \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\") "
Sep 17 19:16:14.909261 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:14.909058    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ad69a7e-adf0-4c53-912b-8f331d91eb43-tmp-dir\") pod \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\" (UID: \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\") "
Sep 17 19:16:14.909261 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:14.909072    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ad69a7e-adf0-4c53-912b-8f331d91eb43-home\") pod \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\" (UID: \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\") "
Sep 17 19:16:14.909261 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:14.909090    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ad69a7e-adf0-4c53-912b-8f331d91eb43-model-cache\") pod \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\" (UID: \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\") "
Sep 17 19:16:14.909261 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:14.909163    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ad69a7e-adf0-4c53-912b-8f331d91eb43-kserve-provision-location\") pod \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\" (UID: \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\") "
Sep 17 19:16:14.909261 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:14.909189    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwvnm\" (UniqueName: \"kubernetes.io/projected/7ad69a7e-adf0-4c53-912b-8f331d91eb43-kube-api-access-jwvnm\") pod \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\" (UID: \"7ad69a7e-adf0-4c53-912b-8f331d91eb43\") "
Sep 17 19:16:14.909620 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:14.909450    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ad69a7e-adf0-4c53-912b-8f331d91eb43-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7ad69a7e-adf0-4c53-912b-8f331d91eb43" (UID: "7ad69a7e-adf0-4c53-912b-8f331d91eb43"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:16:14.909703 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:14.909671    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad69a7e-adf0-4c53-912b-8f331d91eb43-model-cache" (OuterVolumeSpecName: "model-cache") pod "7ad69a7e-adf0-4c53-912b-8f331d91eb43" (UID: "7ad69a7e-adf0-4c53-912b-8f331d91eb43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:16:14.909868 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:14.909831    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad69a7e-adf0-4c53-912b-8f331d91eb43-home" (OuterVolumeSpecName: "home") pod "7ad69a7e-adf0-4c53-912b-8f331d91eb43" (UID: "7ad69a7e-adf0-4c53-912b-8f331d91eb43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:16:14.911437 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:14.911406    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ad69a7e-adf0-4c53-912b-8f331d91eb43-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7ad69a7e-adf0-4c53-912b-8f331d91eb43" (UID: "7ad69a7e-adf0-4c53-912b-8f331d91eb43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:16:14.911694 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:14.911495    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad69a7e-adf0-4c53-912b-8f331d91eb43-dshm" (OuterVolumeSpecName: "dshm") pod "7ad69a7e-adf0-4c53-912b-8f331d91eb43" (UID: "7ad69a7e-adf0-4c53-912b-8f331d91eb43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:16:14.912132 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:14.912107    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad69a7e-adf0-4c53-912b-8f331d91eb43-kube-api-access-jwvnm" (OuterVolumeSpecName: "kube-api-access-jwvnm") pod "7ad69a7e-adf0-4c53-912b-8f331d91eb43" (UID: "7ad69a7e-adf0-4c53-912b-8f331d91eb43"). InnerVolumeSpecName "kube-api-access-jwvnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:16:14.922854 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:14.922828    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad69a7e-adf0-4c53-912b-8f331d91eb43-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7ad69a7e-adf0-4c53-912b-8f331d91eb43" (UID: "7ad69a7e-adf0-4c53-912b-8f331d91eb43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:16:14.931870 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:14.931845    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ad69a7e-adf0-4c53-912b-8f331d91eb43-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7ad69a7e-adf0-4c53-912b-8f331d91eb43" (UID: "7ad69a7e-adf0-4c53-912b-8f331d91eb43"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:16:15.010236 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:15.010149    2588 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ad69a7e-adf0-4c53-912b-8f331d91eb43-dshm\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:16:15.010236 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:15.010179    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ad69a7e-adf0-4c53-912b-8f331d91eb43-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:16:15.010236 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:15.010189    2588 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ad69a7e-adf0-4c53-912b-8f331d91eb43-cabundle-cert\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:16:15.010236 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:15.010197    2588 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ad69a7e-adf0-4c53-912b-8f331d91eb43-tmp-dir\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:16:15.010236 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:15.010205    2588 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ad69a7e-adf0-4c53-912b-8f331d91eb43-home\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:16:15.010236 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:15.010212    2588 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ad69a7e-adf0-4c53-912b-8f331d91eb43-model-cache\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:16:15.010236 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:15.010221    2588 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ad69a7e-adf0-4c53-912b-8f331d91eb43-kserve-provision-location\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:16:15.010236 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:15.010229    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwvnm\" (UniqueName: \"kubernetes.io/projected/7ad69a7e-adf0-4c53-912b-8f331d91eb43-kube-api-access-jwvnm\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:16:15.478103 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:15.478073    2588 generic.go:356] "Generic (PLEG): container finished" podID="7ad69a7e-adf0-4c53-912b-8f331d91eb43" containerID="c543e06f10389665e55d5fac4176c8f9da2b630df33fbb377736f8d453ce27d8" exitCode=0
Sep 17 19:16:15.478103 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:15.478097    2588 generic.go:356] "Generic (PLEG): container finished" podID="7ad69a7e-adf0-4c53-912b-8f331d91eb43" containerID="6caedbde7ee4aac325a6f7df88615c7eebc4ba44e776a3b8c2f4f98ceb1e0d7d" exitCode=0
Sep 17 19:16:15.478316 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:15.478128    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"
Sep 17 19:16:15.478316 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:15.478163    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc" event={"ID":"7ad69a7e-adf0-4c53-912b-8f331d91eb43","Type":"ContainerDied","Data":"c543e06f10389665e55d5fac4176c8f9da2b630df33fbb377736f8d453ce27d8"}
Sep 17 19:16:15.478316 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:15.478223    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc" event={"ID":"7ad69a7e-adf0-4c53-912b-8f331d91eb43","Type":"ContainerDied","Data":"6caedbde7ee4aac325a6f7df88615c7eebc4ba44e776a3b8c2f4f98ceb1e0d7d"}
Sep 17 19:16:15.478316 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:15.478242    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc" event={"ID":"7ad69a7e-adf0-4c53-912b-8f331d91eb43","Type":"ContainerDied","Data":"d547e64342a3e1a22b35dfe37d05c92ba3079bba6e41a3ae2c1f17b4a29000b7"}
Sep 17 19:16:15.478316 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:15.478263    2588 scope.go:117] "RemoveContainer" containerID="c543e06f10389665e55d5fac4176c8f9da2b630df33fbb377736f8d453ce27d8"
Sep 17 19:16:15.493583 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:15.489154    2588 scope.go:117] "RemoveContainer" containerID="3da990122a44913375fe271cefae27305b8f477851273a3915903f146378e30e"
Sep 17 19:16:15.501372 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:15.501348    2588 scope.go:117] "RemoveContainer" containerID="6caedbde7ee4aac325a6f7df88615c7eebc4ba44e776a3b8c2f4f98ceb1e0d7d"
Sep 17 19:16:15.507125 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:15.507100    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"]
Sep 17 19:16:15.510642 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:15.510620    2588 scope.go:117] "RemoveContainer" containerID="c543e06f10389665e55d5fac4176c8f9da2b630df33fbb377736f8d453ce27d8"
Sep 17 19:16:15.510959 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:16:15.510936    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c543e06f10389665e55d5fac4176c8f9da2b630df33fbb377736f8d453ce27d8\": container with ID starting with c543e06f10389665e55d5fac4176c8f9da2b630df33fbb377736f8d453ce27d8 not found: ID does not exist" containerID="c543e06f10389665e55d5fac4176c8f9da2b630df33fbb377736f8d453ce27d8"
Sep 17 19:16:15.511056 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:15.510966    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c543e06f10389665e55d5fac4176c8f9da2b630df33fbb377736f8d453ce27d8"} err="failed to get container status \"c543e06f10389665e55d5fac4176c8f9da2b630df33fbb377736f8d453ce27d8\": rpc error: code = NotFound desc = could not find container \"c543e06f10389665e55d5fac4176c8f9da2b630df33fbb377736f8d453ce27d8\": container with ID starting with c543e06f10389665e55d5fac4176c8f9da2b630df33fbb377736f8d453ce27d8 not found: ID does not exist"
Sep 17 19:16:15.511056 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:15.510984    2588 scope.go:117] "RemoveContainer" containerID="3da990122a44913375fe271cefae27305b8f477851273a3915903f146378e30e"
Sep 17 19:16:15.511286 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:16:15.511267    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3da990122a44913375fe271cefae27305b8f477851273a3915903f146378e30e\": container with ID starting with 3da990122a44913375fe271cefae27305b8f477851273a3915903f146378e30e not found: ID does not exist" containerID="3da990122a44913375fe271cefae27305b8f477851273a3915903f146378e30e"
Sep 17 19:16:15.511337 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:15.511292    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3da990122a44913375fe271cefae27305b8f477851273a3915903f146378e30e"} err="failed to get container status \"3da990122a44913375fe271cefae27305b8f477851273a3915903f146378e30e\": rpc error: code = NotFound desc = could not find container \"3da990122a44913375fe271cefae27305b8f477851273a3915903f146378e30e\": container with ID starting with 3da990122a44913375fe271cefae27305b8f477851273a3915903f146378e30e not found: ID does not exist"
Sep 17 19:16:15.511337 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:15.511308    2588 scope.go:117] "RemoveContainer" containerID="6caedbde7ee4aac325a6f7df88615c7eebc4ba44e776a3b8c2f4f98ceb1e0d7d"
Sep 17 19:16:15.511597 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:16:15.511566    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6caedbde7ee4aac325a6f7df88615c7eebc4ba44e776a3b8c2f4f98ceb1e0d7d\": container with ID starting with 6caedbde7ee4aac325a6f7df88615c7eebc4ba44e776a3b8c2f4f98ceb1e0d7d not found: ID does not exist" containerID="6caedbde7ee4aac325a6f7df88615c7eebc4ba44e776a3b8c2f4f98ceb1e0d7d"
Sep 17 19:16:15.511654 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:15.511604    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6caedbde7ee4aac325a6f7df88615c7eebc4ba44e776a3b8c2f4f98ceb1e0d7d"} err="failed to get container status \"6caedbde7ee4aac325a6f7df88615c7eebc4ba44e776a3b8c2f4f98ceb1e0d7d\": rpc error: code = NotFound desc = could not find container \"6caedbde7ee4aac325a6f7df88615c7eebc4ba44e776a3b8c2f4f98ceb1e0d7d\": container with ID starting with 6caedbde7ee4aac325a6f7df88615c7eebc4ba44e776a3b8c2f4f98ceb1e0d7d not found: ID does not exist"
Sep 17 19:16:15.511654 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:15.511622    2588 scope.go:117] "RemoveContainer" containerID="c543e06f10389665e55d5fac4176c8f9da2b630df33fbb377736f8d453ce27d8"
Sep 17 19:16:15.511757 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:15.511671    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7c7bb9dc78vgsqc"]
Sep 17 19:16:15.511886 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:15.511867    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c543e06f10389665e55d5fac4176c8f9da2b630df33fbb377736f8d453ce27d8"} err="failed to get container status \"c543e06f10389665e55d5fac4176c8f9da2b630df33fbb377736f8d453ce27d8\": rpc error: code = NotFound desc = could not find container \"c543e06f10389665e55d5fac4176c8f9da2b630df33fbb377736f8d453ce27d8\": container with ID starting with c543e06f10389665e55d5fac4176c8f9da2b630df33fbb377736f8d453ce27d8 not found: ID does not exist"
Sep 17 19:16:15.511930 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:15.511887    2588 scope.go:117] "RemoveContainer" containerID="3da990122a44913375fe271cefae27305b8f477851273a3915903f146378e30e"
Sep 17 19:16:15.512112 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:15.512096    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3da990122a44913375fe271cefae27305b8f477851273a3915903f146378e30e"} err="failed to get container status \"3da990122a44913375fe271cefae27305b8f477851273a3915903f146378e30e\": rpc error: code = NotFound desc = could not find container \"3da990122a44913375fe271cefae27305b8f477851273a3915903f146378e30e\": container with ID starting with 3da990122a44913375fe271cefae27305b8f477851273a3915903f146378e30e not found: ID does not exist"
Sep 17 19:16:15.512160 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:15.512113    2588 scope.go:117] "RemoveContainer" containerID="6caedbde7ee4aac325a6f7df88615c7eebc4ba44e776a3b8c2f4f98ceb1e0d7d"
Sep 17 19:16:15.512334 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:15.512316    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6caedbde7ee4aac325a6f7df88615c7eebc4ba44e776a3b8c2f4f98ceb1e0d7d"} err="failed to get container status \"6caedbde7ee4aac325a6f7df88615c7eebc4ba44e776a3b8c2f4f98ceb1e0d7d\": rpc error: code = NotFound desc = could not find container \"6caedbde7ee4aac325a6f7df88615c7eebc4ba44e776a3b8c2f4f98ceb1e0d7d\": container with ID starting with 6caedbde7ee4aac325a6f7df88615c7eebc4ba44e776a3b8c2f4f98ceb1e0d7d not found: ID does not exist"
Sep 17 19:16:17.222812 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:17.222778    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:16:17.505575 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:17.505486    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ad69a7e-adf0-4c53-912b-8f331d91eb43" path="/var/lib/kubelet/pods/7ad69a7e-adf0-4c53-912b-8f331d91eb43/volumes"
Sep 17 19:16:20.538836 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:20.538786    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:16:20.638278 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:20.638249    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:16:21.757720 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:21.757681    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7" podUID="43108c2c-81b9-4a68-af16-35c0648a413e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:16:21.758107 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:21.757776    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7"
Sep 17 19:16:26.886571 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:26.886509    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:16:31.758018 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:31.757979    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7" podUID="43108c2c-81b9-4a68-af16-35c0648a413e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:16:41.758570 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:41.758511    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7" podUID="43108c2c-81b9-4a68-af16-35c0648a413e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:16:44.172051 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:44.172023    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7"
Sep 17 19:16:44.253629 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:44.253508    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x59ws\" (UniqueName: \"kubernetes.io/projected/43108c2c-81b9-4a68-af16-35c0648a413e-kube-api-access-x59ws\") pod \"43108c2c-81b9-4a68-af16-35c0648a413e\" (UID: \"43108c2c-81b9-4a68-af16-35c0648a413e\") "
Sep 17 19:16:44.253629 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:44.253629    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43108c2c-81b9-4a68-af16-35c0648a413e-tls-certs\") pod \"43108c2c-81b9-4a68-af16-35c0648a413e\" (UID: \"43108c2c-81b9-4a68-af16-35c0648a413e\") "
Sep 17 19:16:44.255706 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:44.255673    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43108c2c-81b9-4a68-af16-35c0648a413e-kube-api-access-x59ws" (OuterVolumeSpecName: "kube-api-access-x59ws") pod "43108c2c-81b9-4a68-af16-35c0648a413e" (UID: "43108c2c-81b9-4a68-af16-35c0648a413e"). InnerVolumeSpecName "kube-api-access-x59ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:16:44.255819 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:44.255739    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43108c2c-81b9-4a68-af16-35c0648a413e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "43108c2c-81b9-4a68-af16-35c0648a413e" (UID: "43108c2c-81b9-4a68-af16-35c0648a413e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:16:44.355129 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:44.355080    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x59ws\" (UniqueName: \"kubernetes.io/projected/43108c2c-81b9-4a68-af16-35c0648a413e-kube-api-access-x59ws\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:16:44.355129 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:44.355125    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43108c2c-81b9-4a68-af16-35c0648a413e-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:16:44.590629 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:44.590591    2588 generic.go:356] "Generic (PLEG): container finished" podID="43108c2c-81b9-4a68-af16-35c0648a413e" containerID="5235dd24667f83fea2f0d820518770bd0a81bcb3560c97d7b3c9691c77746e13" exitCode=0
Sep 17 19:16:44.590815 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:44.590633    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7"
Sep 17 19:16:44.590815 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:44.590675    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7" event={"ID":"43108c2c-81b9-4a68-af16-35c0648a413e","Type":"ContainerDied","Data":"5235dd24667f83fea2f0d820518770bd0a81bcb3560c97d7b3c9691c77746e13"}
Sep 17 19:16:44.590815 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:44.590723    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7" event={"ID":"43108c2c-81b9-4a68-af16-35c0648a413e","Type":"ContainerDied","Data":"6de8710114acb98b49e6bcb3060f1b6468c4fa65d9843598aef54cebbbe50124"}
Sep 17 19:16:44.590815 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:44.590745    2588 scope.go:117] "RemoveContainer" containerID="5235dd24667f83fea2f0d820518770bd0a81bcb3560c97d7b3c9691c77746e13"
Sep 17 19:16:44.599461 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:44.599443    2588 scope.go:117] "RemoveContainer" containerID="5235dd24667f83fea2f0d820518770bd0a81bcb3560c97d7b3c9691c77746e13"
Sep 17 19:16:44.599761 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:16:44.599744    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5235dd24667f83fea2f0d820518770bd0a81bcb3560c97d7b3c9691c77746e13\": container with ID starting with 5235dd24667f83fea2f0d820518770bd0a81bcb3560c97d7b3c9691c77746e13 not found: ID does not exist" containerID="5235dd24667f83fea2f0d820518770bd0a81bcb3560c97d7b3c9691c77746e13"
Sep 17 19:16:44.599825 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:44.599775    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5235dd24667f83fea2f0d820518770bd0a81bcb3560c97d7b3c9691c77746e13"} err="failed to get container status \"5235dd24667f83fea2f0d820518770bd0a81bcb3560c97d7b3c9691c77746e13\": rpc error: code = NotFound desc = could not find container \"5235dd24667f83fea2f0d820518770bd0a81bcb3560c97d7b3c9691c77746e13\": container with ID starting with 5235dd24667f83fea2f0d820518770bd0a81bcb3560c97d7b3c9691c77746e13 not found: ID does not exist"
Sep 17 19:16:44.612836 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:44.612809    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7"]
Sep 17 19:16:44.617513 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:44.617491    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche54pq7"]
Sep 17 19:16:45.505522 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:45.505488    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43108c2c-81b9-4a68-af16-35c0648a413e" path="/var/lib/kubelet/pods/43108c2c-81b9-4a68-af16-35c0648a413e/volumes"
Sep 17 19:16:49.028612 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:49.028575    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:16:49.124330 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:49.124302    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:16:54.869558 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:54.869505    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"]
Sep 17 19:16:54.869981 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:54.869811    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17e5616e-4770-4b2d-98e5-ce5426e1e760" containerName="main"
Sep 17 19:16:54.869981 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:54.869822    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="17e5616e-4770-4b2d-98e5-ce5426e1e760" containerName="main"
Sep 17 19:16:54.869981 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:54.869833    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ad69a7e-adf0-4c53-912b-8f331d91eb43" containerName="main"
Sep 17 19:16:54.869981 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:54.869839    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ad69a7e-adf0-4c53-912b-8f331d91eb43" containerName="main"
Sep 17 19:16:54.869981 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:54.869847    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43108c2c-81b9-4a68-af16-35c0648a413e" containerName="main"
Sep 17 19:16:54.869981 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:54.869854    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="43108c2c-81b9-4a68-af16-35c0648a413e" containerName="main"
Sep 17 19:16:54.869981 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:54.869863    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ad69a7e-adf0-4c53-912b-8f331d91eb43" containerName="llm-d-routing-sidecar"
Sep 17 19:16:54.869981 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:54.869868    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ad69a7e-adf0-4c53-912b-8f331d91eb43" containerName="llm-d-routing-sidecar"
Sep 17 19:16:54.869981 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:54.869875    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ad69a7e-adf0-4c53-912b-8f331d91eb43" containerName="storage-initializer"
Sep 17 19:16:54.869981 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:54.869882    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ad69a7e-adf0-4c53-912b-8f331d91eb43" containerName="storage-initializer"
Sep 17 19:16:54.869981 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:54.869931    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="17e5616e-4770-4b2d-98e5-ce5426e1e760" containerName="main"
Sep 17 19:16:54.869981 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:54.869938    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="43108c2c-81b9-4a68-af16-35c0648a413e" containerName="main"
Sep 17 19:16:54.869981 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:54.869943    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ad69a7e-adf0-4c53-912b-8f331d91eb43" containerName="llm-d-routing-sidecar"
Sep 17 19:16:54.869981 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:54.869948    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ad69a7e-adf0-4c53-912b-8f331d91eb43" containerName="main"
Sep 17 19:16:54.872775 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:54.872749    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:16:54.875263 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:54.875237    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 17 19:16:54.875428 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:54.875409    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 17 19:16:54.875749 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:54.875266    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:16:54.875859 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:54.875491    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 17 19:16:54.875995 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:54.875734    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-5kpg5\""
Sep 17 19:16:54.876879 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:54.876855    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-pzwrj\""
Sep 17 19:16:54.876977 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:54.876884    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 17 19:16:54.891300 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:54.891266    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"]
Sep 17 19:16:54.933730 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:54.933697    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:16:54.935690 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:54.935668    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aef7a70c-cafb-48cd-9ecb-9a729369fb90-model-cache\") pod \"custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn\" (UID: \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:16:54.935786 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:54.935701    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aef7a70c-cafb-48cd-9ecb-9a729369fb90-home\") pod \"custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn\" (UID: \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:16:54.935786 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:54.935718    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aef7a70c-cafb-48cd-9ecb-9a729369fb90-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn\" (UID: \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:16:54.935786 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:54.935749    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aef7a70c-cafb-48cd-9ecb-9a729369fb90-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn\" (UID: \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:16:54.935900 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:54.935795    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aef7a70c-cafb-48cd-9ecb-9a729369fb90-dshm\") pod \"custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn\" (UID: \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:16:54.935900 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:54.935816    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aef7a70c-cafb-48cd-9ecb-9a729369fb90-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn\" (UID: \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:16:54.935900 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:54.935840    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cpr8\" (UniqueName: \"kubernetes.io/projected/aef7a70c-cafb-48cd-9ecb-9a729369fb90-kube-api-access-7cpr8\") pod \"custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn\" (UID: \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:16:54.935900 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:54.935866    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aef7a70c-cafb-48cd-9ecb-9a729369fb90-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn\" (UID: \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:16:55.036331 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.036300    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aef7a70c-cafb-48cd-9ecb-9a729369fb90-home\") pod \"custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn\" (UID: \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:16:55.036331 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.036331    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aef7a70c-cafb-48cd-9ecb-9a729369fb90-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn\" (UID: \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:16:55.036583 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.036367    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aef7a70c-cafb-48cd-9ecb-9a729369fb90-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn\" (UID: \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:16:55.036583 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.036410    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aef7a70c-cafb-48cd-9ecb-9a729369fb90-dshm\") pod \"custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn\" (UID: \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:16:55.036583 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.036445    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aef7a70c-cafb-48cd-9ecb-9a729369fb90-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn\" (UID: \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:16:55.036583 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.036469    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cpr8\" (UniqueName: \"kubernetes.io/projected/aef7a70c-cafb-48cd-9ecb-9a729369fb90-kube-api-access-7cpr8\") pod \"custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn\" (UID: \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:16:55.036850 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.036651    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aef7a70c-cafb-48cd-9ecb-9a729369fb90-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn\" (UID: \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:16:55.036850 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.036724    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aef7a70c-cafb-48cd-9ecb-9a729369fb90-model-cache\") pod \"custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn\" (UID: \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:16:55.036850 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.036776    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aef7a70c-cafb-48cd-9ecb-9a729369fb90-home\") pod \"custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn\" (UID: \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:16:55.037036 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.036964    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aef7a70c-cafb-48cd-9ecb-9a729369fb90-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn\" (UID: \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:16:55.037097 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.037068    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aef7a70c-cafb-48cd-9ecb-9a729369fb90-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn\" (UID: \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:16:55.037139 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.037091    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aef7a70c-cafb-48cd-9ecb-9a729369fb90-model-cache\") pod \"custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn\" (UID: \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:16:55.037224 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.037208    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aef7a70c-cafb-48cd-9ecb-9a729369fb90-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn\" (UID: \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:16:55.038847 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.038819    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aef7a70c-cafb-48cd-9ecb-9a729369fb90-dshm\") pod \"custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn\" (UID: \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:16:55.039016 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.039000    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aef7a70c-cafb-48cd-9ecb-9a729369fb90-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn\" (UID: \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:16:55.045140 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.045118    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cpr8\" (UniqueName: \"kubernetes.io/projected/aef7a70c-cafb-48cd-9ecb-9a729369fb90-kube-api-access-7cpr8\") pod \"custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn\" (UID: \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:16:55.111030 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.110995    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m"]
Sep 17 19:16:55.114978 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.114959    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m"
Sep 17 19:16:55.117693 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.117664    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-6cfvw\""
Sep 17 19:16:55.125481 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.125458    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m"]
Sep 17 19:16:55.137691 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.137664    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62c935a0-e5c3-42f8-abd7-3b6115d4895d-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m\" (UID: \"62c935a0-e5c3-42f8-abd7-3b6115d4895d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m"
Sep 17 19:16:55.137817 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.137715    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsxcq\" (UniqueName: \"kubernetes.io/projected/62c935a0-e5c3-42f8-abd7-3b6115d4895d-kube-api-access-xsxcq\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m\" (UID: \"62c935a0-e5c3-42f8-abd7-3b6115d4895d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m"
Sep 17 19:16:55.183154 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.183116    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:16:55.200475 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.200451    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:16:55.238985 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.238954    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62c935a0-e5c3-42f8-abd7-3b6115d4895d-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m\" (UID: \"62c935a0-e5c3-42f8-abd7-3b6115d4895d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m"
Sep 17 19:16:55.239117 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.239014    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsxcq\" (UniqueName: \"kubernetes.io/projected/62c935a0-e5c3-42f8-abd7-3b6115d4895d-kube-api-access-xsxcq\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m\" (UID: \"62c935a0-e5c3-42f8-abd7-3b6115d4895d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m"
Sep 17 19:16:55.241629 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.241606    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62c935a0-e5c3-42f8-abd7-3b6115d4895d-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m\" (UID: \"62c935a0-e5c3-42f8-abd7-3b6115d4895d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m"
Sep 17 19:16:55.248469 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.248416    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsxcq\" (UniqueName: \"kubernetes.io/projected/62c935a0-e5c3-42f8-abd7-3b6115d4895d-kube-api-access-xsxcq\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m\" (UID: \"62c935a0-e5c3-42f8-abd7-3b6115d4895d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m"
Sep 17 19:16:55.321787 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.321751    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:16:55.425406 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.425307    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m"
Sep 17 19:16:55.465949 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.465913    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"]
Sep 17 19:16:55.469800 ip-10-0-136-132 kubenswrapper[2588]: W0917 19:16:55.469763    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaef7a70c_cafb_48cd_9ecb_9a729369fb90.slice/crio-8a529e55d0f9e9d65a38ded5084c78ffb4972929f9f158b42cf12e1c4d2ec6fa WatchSource:0}: Error finding container 8a529e55d0f9e9d65a38ded5084c78ffb4972929f9f158b42cf12e1c4d2ec6fa: Status 404 returned error can't find the container with id 8a529e55d0f9e9d65a38ded5084c78ffb4972929f9f158b42cf12e1c4d2ec6fa
Sep 17 19:16:55.637560 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.637512    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn" event={"ID":"aef7a70c-cafb-48cd-9ecb-9a729369fb90","Type":"ContainerStarted","Data":"fca8611173cec2c3bb45bdd8f2b5e7895ed773cd53bb5b2bf6c46e9a40e97ee7"}
Sep 17 19:16:55.637725 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.637580    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn" event={"ID":"aef7a70c-cafb-48cd-9ecb-9a729369fb90","Type":"ContainerStarted","Data":"8a529e55d0f9e9d65a38ded5084c78ffb4972929f9f158b42cf12e1c4d2ec6fa"}
Sep 17 19:16:55.637725 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.637660    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:16:55.703229 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:55.703202    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m"]
Sep 17 19:16:55.706266 ip-10-0-136-132 kubenswrapper[2588]: W0917 19:16:55.706234    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c935a0_e5c3_42f8_abd7_3b6115d4895d.slice/crio-64733a752e8a50bd9ca932c6983b9f2c4a83f4a94d24ac0fd2f9150278009c17 WatchSource:0}: Error finding container 64733a752e8a50bd9ca932c6983b9f2c4a83f4a94d24ac0fd2f9150278009c17: Status 404 returned error can't find the container with id 64733a752e8a50bd9ca932c6983b9f2c4a83f4a94d24ac0fd2f9150278009c17
Sep 17 19:16:56.643497 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:56.643459    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn" event={"ID":"aef7a70c-cafb-48cd-9ecb-9a729369fb90","Type":"ContainerStarted","Data":"6a9fab2040014c8c361d765acbced808e998820b414b26f7632a2adf68d204c6"}
Sep 17 19:16:56.650916 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:56.650880    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m" event={"ID":"62c935a0-e5c3-42f8-abd7-3b6115d4895d","Type":"ContainerStarted","Data":"b9bc1c92e25b49c12ca62694bdd54bf5be19df2681ac5d74d105029833d1e782"}
Sep 17 19:16:56.651066 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:56.650968    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m" event={"ID":"62c935a0-e5c3-42f8-abd7-3b6115d4895d","Type":"ContainerStarted","Data":"64733a752e8a50bd9ca932c6983b9f2c4a83f4a94d24ac0fd2f9150278009c17"}
Sep 17 19:16:56.651066 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:56.650992    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m"
Sep 17 19:16:56.685188 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:56.685140    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m" podStartSLOduration=1.685125537 podStartE2EDuration="1.685125537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:16:56.683799032 +0000 UTC m=+2853.534804462" watchObservedRunningTime="2026-09-17 19:16:56.685125537 +0000 UTC m=+2853.536130946"
Sep 17 19:16:58.662598 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:58.662565    2588 generic.go:356] "Generic (PLEG): container finished" podID="aef7a70c-cafb-48cd-9ecb-9a729369fb90" containerID="6a9fab2040014c8c361d765acbced808e998820b414b26f7632a2adf68d204c6" exitCode=0
Sep 17 19:16:58.662989 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:58.662640    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn" event={"ID":"aef7a70c-cafb-48cd-9ecb-9a729369fb90","Type":"ContainerDied","Data":"6a9fab2040014c8c361d765acbced808e998820b414b26f7632a2adf68d204c6"}
Sep 17 19:16:59.670369 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:59.670333    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn" event={"ID":"aef7a70c-cafb-48cd-9ecb-9a729369fb90","Type":"ContainerStarted","Data":"762157e203d8d4d0fdf2144c472784db23c68f71273601e1ecbcb931b5e7c5ff"}
Sep 17 19:16:59.716810 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:16:59.716713    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn" podStartSLOduration=3.5417508829999997 podStartE2EDuration="5.716691763s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.17494088s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:16:59.714086703 +0000 UTC m=+2856.565092113" watchObservedRunningTime="2026-09-17 19:16:59.716691763 +0000 UTC m=+2856.567697165"
Sep 17 19:17:01.070347 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:17:01.070305    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:17:01.169703 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:17:01.169668    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:17:05.183234 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:17:05.183193    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:17:05.183871 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:17:05.183262    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:17:05.184568 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:17:05.184509    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn" podUID="aef7a70c-cafb-48cd-9ecb-9a729369fb90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Sep 17 19:17:05.196184 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:17:05.196157    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:17:09.016201 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:17:09.016095    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:17:15.183983 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:17:15.183939    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn" podUID="aef7a70c-cafb-48cd-9ecb-9a729369fb90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Sep 17 19:17:25.184434 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:17:25.184301    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn" podUID="aef7a70c-cafb-48cd-9ecb-9a729369fb90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Sep 17 19:17:27.657625 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:17:27.657592    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m"
Sep 17 19:17:35.184520 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:17:35.184466    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn" podUID="aef7a70c-cafb-48cd-9ecb-9a729369fb90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Sep 17 19:17:45.184112 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:17:45.184066    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn" podUID="aef7a70c-cafb-48cd-9ecb-9a729369fb90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Sep 17 19:17:55.183789 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:17:55.183735    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn" podUID="aef7a70c-cafb-48cd-9ecb-9a729369fb90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Sep 17 19:18:05.183725 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:18:05.183676    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn" podUID="aef7a70c-cafb-48cd-9ecb-9a729369fb90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Sep 17 19:18:15.184294 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:18:15.184237    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn" podUID="aef7a70c-cafb-48cd-9ecb-9a729369fb90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Sep 17 19:18:17.212809 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:18:17.212766    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:18:25.184272 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:18:25.184218    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn" podUID="aef7a70c-cafb-48cd-9ecb-9a729369fb90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Sep 17 19:18:35.183941 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:18:35.183878    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn" podUID="aef7a70c-cafb-48cd-9ecb-9a729369fb90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Sep 17 19:18:37.200016 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:18:37.199929    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:18:37.316504 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:18:37.316463    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:18:37.416339 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:18:37.416300    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:18:39.009809 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:18:39.009731    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:18:43.586715 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:18:43.586677    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:18:43.706409 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:18:43.706377    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:18:45.184073 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:18:45.184024    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn" podUID="aef7a70c-cafb-48cd-9ecb-9a729369fb90" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Sep 17 19:18:45.718610 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:18:45.718575    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:18:55.193265 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:18:55.193232    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:18:55.209631 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:18:55.209601    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:19:11.281830 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:11.281796    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m"]
Sep 17 19:19:11.286391 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:11.286359    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"]
Sep 17 19:19:11.298205 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:11.298171    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:19:11.406020 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:11.405979    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:19:17.657804 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:17.657764    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m" podUID="62c935a0-e5c3-42f8-abd7-3b6115d4895d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:19:25.684872 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:25.684824    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xlklb_07ee31e0-43bb-4940-837d-969bd840492b/console-operator/1.log"
Sep 17 19:19:25.688446 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:25.688424    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xlklb_07ee31e0-43bb-4940-837d-969bd840492b/console-operator/1.log"
Sep 17 19:19:25.690395 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:25.690375    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovn-acl-logging/0.log"
Sep 17 19:19:25.693744 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:25.693727    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovn-acl-logging/0.log"
Sep 17 19:19:26.308886 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:26.308826    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn" podUID="aef7a70c-cafb-48cd-9ecb-9a729369fb90" containerName="main" containerID="cri-o://762157e203d8d4d0fdf2144c472784db23c68f71273601e1ecbcb931b5e7c5ff" gracePeriod=45
Sep 17 19:19:26.310332 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:26.310286    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m" podUID="62c935a0-e5c3-42f8-abd7-3b6115d4895d" containerName="main" containerID="cri-o://b9bc1c92e25b49c12ca62694bdd54bf5be19df2681ac5d74d105029833d1e782" gracePeriod=45
Sep 17 19:19:26.925761 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:26.925698    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn" podUID="aef7a70c-cafb-48cd-9ecb-9a729369fb90" containerName="llm-d-routing-sidecar" containerID="cri-o://fca8611173cec2c3bb45bdd8f2b5e7895ed773cd53bb5b2bf6c46e9a40e97ee7" gracePeriod=45
Sep 17 19:19:27.173267 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.173241    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:19:27.315084 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.315044    2588 generic.go:356] "Generic (PLEG): container finished" podID="aef7a70c-cafb-48cd-9ecb-9a729369fb90" containerID="762157e203d8d4d0fdf2144c472784db23c68f71273601e1ecbcb931b5e7c5ff" exitCode=0
Sep 17 19:19:27.315084 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.315072    2588 generic.go:356] "Generic (PLEG): container finished" podID="aef7a70c-cafb-48cd-9ecb-9a729369fb90" containerID="fca8611173cec2c3bb45bdd8f2b5e7895ed773cd53bb5b2bf6c46e9a40e97ee7" exitCode=0
Sep 17 19:19:27.315310 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.315104    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"
Sep 17 19:19:27.315310 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.315128    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn" event={"ID":"aef7a70c-cafb-48cd-9ecb-9a729369fb90","Type":"ContainerDied","Data":"762157e203d8d4d0fdf2144c472784db23c68f71273601e1ecbcb931b5e7c5ff"}
Sep 17 19:19:27.315310 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.315215    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn" event={"ID":"aef7a70c-cafb-48cd-9ecb-9a729369fb90","Type":"ContainerDied","Data":"fca8611173cec2c3bb45bdd8f2b5e7895ed773cd53bb5b2bf6c46e9a40e97ee7"}
Sep 17 19:19:27.315310 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.315236    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn" event={"ID":"aef7a70c-cafb-48cd-9ecb-9a729369fb90","Type":"ContainerDied","Data":"8a529e55d0f9e9d65a38ded5084c78ffb4972929f9f158b42cf12e1c4d2ec6fa"}
Sep 17 19:19:27.315310 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.315263    2588 scope.go:117] "RemoveContainer" containerID="762157e203d8d4d0fdf2144c472784db23c68f71273601e1ecbcb931b5e7c5ff"
Sep 17 19:19:27.323610 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.323592    2588 scope.go:117] "RemoveContainer" containerID="6a9fab2040014c8c361d765acbced808e998820b414b26f7632a2adf68d204c6"
Sep 17 19:19:27.334160 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.334133    2588 scope.go:117] "RemoveContainer" containerID="fca8611173cec2c3bb45bdd8f2b5e7895ed773cd53bb5b2bf6c46e9a40e97ee7"
Sep 17 19:19:27.342140 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.342111    2588 scope.go:117] "RemoveContainer" containerID="762157e203d8d4d0fdf2144c472784db23c68f71273601e1ecbcb931b5e7c5ff"
Sep 17 19:19:27.342405 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:19:27.342386    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"762157e203d8d4d0fdf2144c472784db23c68f71273601e1ecbcb931b5e7c5ff\": container with ID starting with 762157e203d8d4d0fdf2144c472784db23c68f71273601e1ecbcb931b5e7c5ff not found: ID does not exist" containerID="762157e203d8d4d0fdf2144c472784db23c68f71273601e1ecbcb931b5e7c5ff"
Sep 17 19:19:27.342459 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.342416    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"762157e203d8d4d0fdf2144c472784db23c68f71273601e1ecbcb931b5e7c5ff"} err="failed to get container status \"762157e203d8d4d0fdf2144c472784db23c68f71273601e1ecbcb931b5e7c5ff\": rpc error: code = NotFound desc = could not find container \"762157e203d8d4d0fdf2144c472784db23c68f71273601e1ecbcb931b5e7c5ff\": container with ID starting with 762157e203d8d4d0fdf2144c472784db23c68f71273601e1ecbcb931b5e7c5ff not found: ID does not exist"
Sep 17 19:19:27.342459 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.342435    2588 scope.go:117] "RemoveContainer" containerID="6a9fab2040014c8c361d765acbced808e998820b414b26f7632a2adf68d204c6"
Sep 17 19:19:27.342756 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:19:27.342736    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a9fab2040014c8c361d765acbced808e998820b414b26f7632a2adf68d204c6\": container with ID starting with 6a9fab2040014c8c361d765acbced808e998820b414b26f7632a2adf68d204c6 not found: ID does not exist" containerID="6a9fab2040014c8c361d765acbced808e998820b414b26f7632a2adf68d204c6"
Sep 17 19:19:27.342806 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.342765    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a9fab2040014c8c361d765acbced808e998820b414b26f7632a2adf68d204c6"} err="failed to get container status \"6a9fab2040014c8c361d765acbced808e998820b414b26f7632a2adf68d204c6\": rpc error: code = NotFound desc = could not find container \"6a9fab2040014c8c361d765acbced808e998820b414b26f7632a2adf68d204c6\": container with ID starting with 6a9fab2040014c8c361d765acbced808e998820b414b26f7632a2adf68d204c6 not found: ID does not exist"
Sep 17 19:19:27.342806 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.342782    2588 scope.go:117] "RemoveContainer" containerID="fca8611173cec2c3bb45bdd8f2b5e7895ed773cd53bb5b2bf6c46e9a40e97ee7"
Sep 17 19:19:27.343020 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:19:27.343003    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fca8611173cec2c3bb45bdd8f2b5e7895ed773cd53bb5b2bf6c46e9a40e97ee7\": container with ID starting with fca8611173cec2c3bb45bdd8f2b5e7895ed773cd53bb5b2bf6c46e9a40e97ee7 not found: ID does not exist" containerID="fca8611173cec2c3bb45bdd8f2b5e7895ed773cd53bb5b2bf6c46e9a40e97ee7"
Sep 17 19:19:27.343063 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.343024    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca8611173cec2c3bb45bdd8f2b5e7895ed773cd53bb5b2bf6c46e9a40e97ee7"} err="failed to get container status \"fca8611173cec2c3bb45bdd8f2b5e7895ed773cd53bb5b2bf6c46e9a40e97ee7\": rpc error: code = NotFound desc = could not find container \"fca8611173cec2c3bb45bdd8f2b5e7895ed773cd53bb5b2bf6c46e9a40e97ee7\": container with ID starting with fca8611173cec2c3bb45bdd8f2b5e7895ed773cd53bb5b2bf6c46e9a40e97ee7 not found: ID does not exist"
Sep 17 19:19:27.343063 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.343039    2588 scope.go:117] "RemoveContainer" containerID="762157e203d8d4d0fdf2144c472784db23c68f71273601e1ecbcb931b5e7c5ff"
Sep 17 19:19:27.343276 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.343255    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"762157e203d8d4d0fdf2144c472784db23c68f71273601e1ecbcb931b5e7c5ff"} err="failed to get container status \"762157e203d8d4d0fdf2144c472784db23c68f71273601e1ecbcb931b5e7c5ff\": rpc error: code = NotFound desc = could not find container \"762157e203d8d4d0fdf2144c472784db23c68f71273601e1ecbcb931b5e7c5ff\": container with ID starting with 762157e203d8d4d0fdf2144c472784db23c68f71273601e1ecbcb931b5e7c5ff not found: ID does not exist"
Sep 17 19:19:27.343361 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.343278    2588 scope.go:117] "RemoveContainer" containerID="6a9fab2040014c8c361d765acbced808e998820b414b26f7632a2adf68d204c6"
Sep 17 19:19:27.343559 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.343515    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a9fab2040014c8c361d765acbced808e998820b414b26f7632a2adf68d204c6"} err="failed to get container status \"6a9fab2040014c8c361d765acbced808e998820b414b26f7632a2adf68d204c6\": rpc error: code = NotFound desc = could not find container \"6a9fab2040014c8c361d765acbced808e998820b414b26f7632a2adf68d204c6\": container with ID starting with 6a9fab2040014c8c361d765acbced808e998820b414b26f7632a2adf68d204c6 not found: ID does not exist"
Sep 17 19:19:27.343622 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.343562    2588 scope.go:117] "RemoveContainer" containerID="fca8611173cec2c3bb45bdd8f2b5e7895ed773cd53bb5b2bf6c46e9a40e97ee7"
Sep 17 19:19:27.343822 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.343805    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca8611173cec2c3bb45bdd8f2b5e7895ed773cd53bb5b2bf6c46e9a40e97ee7"} err="failed to get container status \"fca8611173cec2c3bb45bdd8f2b5e7895ed773cd53bb5b2bf6c46e9a40e97ee7\": rpc error: code = NotFound desc = could not find container \"fca8611173cec2c3bb45bdd8f2b5e7895ed773cd53bb5b2bf6c46e9a40e97ee7\": container with ID starting with fca8611173cec2c3bb45bdd8f2b5e7895ed773cd53bb5b2bf6c46e9a40e97ee7 not found: ID does not exist"
Sep 17 19:19:27.350095 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.350077    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aef7a70c-cafb-48cd-9ecb-9a729369fb90-home\") pod \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\" (UID: \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\") "
Sep 17 19:19:27.350182 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.350104    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aef7a70c-cafb-48cd-9ecb-9a729369fb90-cabundle-cert\") pod \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\" (UID: \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\") "
Sep 17 19:19:27.350182 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.350151    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aef7a70c-cafb-48cd-9ecb-9a729369fb90-model-cache\") pod \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\" (UID: \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\") "
Sep 17 19:19:27.350182 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.350176    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aef7a70c-cafb-48cd-9ecb-9a729369fb90-dshm\") pod \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\" (UID: \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\") "
Sep 17 19:19:27.350339 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.350193    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cpr8\" (UniqueName: \"kubernetes.io/projected/aef7a70c-cafb-48cd-9ecb-9a729369fb90-kube-api-access-7cpr8\") pod \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\" (UID: \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\") "
Sep 17 19:19:27.350339 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.350213    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aef7a70c-cafb-48cd-9ecb-9a729369fb90-kserve-provision-location\") pod \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\" (UID: \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\") "
Sep 17 19:19:27.350339 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.350244    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aef7a70c-cafb-48cd-9ecb-9a729369fb90-tls-certs\") pod \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\" (UID: \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\") "
Sep 17 19:19:27.350339 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.350274    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aef7a70c-cafb-48cd-9ecb-9a729369fb90-tmp-dir\") pod \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\" (UID: \"aef7a70c-cafb-48cd-9ecb-9a729369fb90\") "
Sep 17 19:19:27.350625 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.350477    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aef7a70c-cafb-48cd-9ecb-9a729369fb90-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "aef7a70c-cafb-48cd-9ecb-9a729369fb90" (UID: "aef7a70c-cafb-48cd-9ecb-9a729369fb90"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:19:27.350746 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.350723    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef7a70c-cafb-48cd-9ecb-9a729369fb90-model-cache" (OuterVolumeSpecName: "model-cache") pod "aef7a70c-cafb-48cd-9ecb-9a729369fb90" (UID: "aef7a70c-cafb-48cd-9ecb-9a729369fb90"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:19:27.350803 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.350786    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef7a70c-cafb-48cd-9ecb-9a729369fb90-home" (OuterVolumeSpecName: "home") pod "aef7a70c-cafb-48cd-9ecb-9a729369fb90" (UID: "aef7a70c-cafb-48cd-9ecb-9a729369fb90"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:19:27.353020 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.352990    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aef7a70c-cafb-48cd-9ecb-9a729369fb90-kube-api-access-7cpr8" (OuterVolumeSpecName: "kube-api-access-7cpr8") pod "aef7a70c-cafb-48cd-9ecb-9a729369fb90" (UID: "aef7a70c-cafb-48cd-9ecb-9a729369fb90"). InnerVolumeSpecName "kube-api-access-7cpr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:19:27.353234 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.353212    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef7a70c-cafb-48cd-9ecb-9a729369fb90-dshm" (OuterVolumeSpecName: "dshm") pod "aef7a70c-cafb-48cd-9ecb-9a729369fb90" (UID: "aef7a70c-cafb-48cd-9ecb-9a729369fb90"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:19:27.353234 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.353216    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aef7a70c-cafb-48cd-9ecb-9a729369fb90-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aef7a70c-cafb-48cd-9ecb-9a729369fb90" (UID: "aef7a70c-cafb-48cd-9ecb-9a729369fb90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:19:27.364643 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.364607    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef7a70c-cafb-48cd-9ecb-9a729369fb90-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aef7a70c-cafb-48cd-9ecb-9a729369fb90" (UID: "aef7a70c-cafb-48cd-9ecb-9a729369fb90"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:19:27.376736 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.376709    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef7a70c-cafb-48cd-9ecb-9a729369fb90-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "aef7a70c-cafb-48cd-9ecb-9a729369fb90" (UID: "aef7a70c-cafb-48cd-9ecb-9a729369fb90"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:19:27.451389 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.451350    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aef7a70c-cafb-48cd-9ecb-9a729369fb90-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:19:27.451633 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.451404    2588 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aef7a70c-cafb-48cd-9ecb-9a729369fb90-tmp-dir\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:19:27.451633 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.451413    2588 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aef7a70c-cafb-48cd-9ecb-9a729369fb90-home\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:19:27.451633 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.451422    2588 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aef7a70c-cafb-48cd-9ecb-9a729369fb90-cabundle-cert\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:19:27.451633 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.451430    2588 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aef7a70c-cafb-48cd-9ecb-9a729369fb90-model-cache\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:19:27.451633 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.451438    2588 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aef7a70c-cafb-48cd-9ecb-9a729369fb90-dshm\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:19:27.451633 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.451445    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cpr8\" (UniqueName: \"kubernetes.io/projected/aef7a70c-cafb-48cd-9ecb-9a729369fb90-kube-api-access-7cpr8\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:19:27.451633 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.451455    2588 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aef7a70c-cafb-48cd-9ecb-9a729369fb90-kserve-provision-location\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:19:27.634154 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.634070    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"]
Sep 17 19:19:27.638586 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.638553    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77db7c8b7-g5bsn"]
Sep 17 19:19:27.657636 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:27.657601    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m" podUID="62c935a0-e5c3-42f8-abd7-3b6115d4895d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:19:29.506462 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:29.506426    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aef7a70c-cafb-48cd-9ecb-9a729369fb90" path="/var/lib/kubelet/pods/aef7a70c-cafb-48cd-9ecb-9a729369fb90/volumes"
Sep 17 19:19:37.658202 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:37.658162    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m" podUID="62c935a0-e5c3-42f8-abd7-3b6115d4895d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:19:37.658686 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:37.658273    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m"
Sep 17 19:19:47.658476 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:47.658435    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m" podUID="62c935a0-e5c3-42f8-abd7-3b6115d4895d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:19:56.433067 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:56.433032    2588 generic.go:356] "Generic (PLEG): container finished" podID="62c935a0-e5c3-42f8-abd7-3b6115d4895d" containerID="b9bc1c92e25b49c12ca62694bdd54bf5be19df2681ac5d74d105029833d1e782" exitCode=0
Sep 17 19:19:56.433388 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:56.433083    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m" event={"ID":"62c935a0-e5c3-42f8-abd7-3b6115d4895d","Type":"ContainerDied","Data":"b9bc1c92e25b49c12ca62694bdd54bf5be19df2681ac5d74d105029833d1e782"}
Sep 17 19:19:56.564777 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:56.564750    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m"
Sep 17 19:19:56.687091 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:56.686984    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62c935a0-e5c3-42f8-abd7-3b6115d4895d-tls-certs\") pod \"62c935a0-e5c3-42f8-abd7-3b6115d4895d\" (UID: \"62c935a0-e5c3-42f8-abd7-3b6115d4895d\") "
Sep 17 19:19:56.687278 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:56.687104    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsxcq\" (UniqueName: \"kubernetes.io/projected/62c935a0-e5c3-42f8-abd7-3b6115d4895d-kube-api-access-xsxcq\") pod \"62c935a0-e5c3-42f8-abd7-3b6115d4895d\" (UID: \"62c935a0-e5c3-42f8-abd7-3b6115d4895d\") "
Sep 17 19:19:56.689179 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:56.689139    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c935a0-e5c3-42f8-abd7-3b6115d4895d-kube-api-access-xsxcq" (OuterVolumeSpecName: "kube-api-access-xsxcq") pod "62c935a0-e5c3-42f8-abd7-3b6115d4895d" (UID: "62c935a0-e5c3-42f8-abd7-3b6115d4895d"). InnerVolumeSpecName "kube-api-access-xsxcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:19:56.689179 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:56.689161    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62c935a0-e5c3-42f8-abd7-3b6115d4895d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "62c935a0-e5c3-42f8-abd7-3b6115d4895d" (UID: "62c935a0-e5c3-42f8-abd7-3b6115d4895d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:19:56.788712 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:56.788671    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsxcq\" (UniqueName: \"kubernetes.io/projected/62c935a0-e5c3-42f8-abd7-3b6115d4895d-kube-api-access-xsxcq\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:19:56.788712 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:56.788703    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62c935a0-e5c3-42f8-abd7-3b6115d4895d-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:19:57.438796 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:57.438761    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m"
Sep 17 19:19:57.439246 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:57.438795    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m" event={"ID":"62c935a0-e5c3-42f8-abd7-3b6115d4895d","Type":"ContainerDied","Data":"64733a752e8a50bd9ca932c6983b9f2c4a83f4a94d24ac0fd2f9150278009c17"}
Sep 17 19:19:57.439246 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:57.438856    2588 scope.go:117] "RemoveContainer" containerID="b9bc1c92e25b49c12ca62694bdd54bf5be19df2681ac5d74d105029833d1e782"
Sep 17 19:19:57.461266 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:57.461240    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m"]
Sep 17 19:19:57.465218 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:57.465191    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48jl59m"]
Sep 17 19:19:57.505865 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:19:57.505836    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62c935a0-e5c3-42f8-abd7-3b6115d4895d" path="/var/lib/kubelet/pods/62c935a0-e5c3-42f8-abd7-3b6115d4895d/volumes"
Sep 17 19:20:01.370539 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.370504    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:01.466322 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.466280    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:01.618823 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.618791    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"]
Sep 17 19:20:01.619148 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.619131    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aef7a70c-cafb-48cd-9ecb-9a729369fb90" containerName="storage-initializer"
Sep 17 19:20:01.619228 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.619151    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="aef7a70c-cafb-48cd-9ecb-9a729369fb90" containerName="storage-initializer"
Sep 17 19:20:01.619228 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.619168    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aef7a70c-cafb-48cd-9ecb-9a729369fb90" containerName="main"
Sep 17 19:20:01.619228 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.619176    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="aef7a70c-cafb-48cd-9ecb-9a729369fb90" containerName="main"
Sep 17 19:20:01.619228 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.619197    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aef7a70c-cafb-48cd-9ecb-9a729369fb90" containerName="llm-d-routing-sidecar"
Sep 17 19:20:01.619228 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.619206    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="aef7a70c-cafb-48cd-9ecb-9a729369fb90" containerName="llm-d-routing-sidecar"
Sep 17 19:20:01.619228 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.619220    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62c935a0-e5c3-42f8-abd7-3b6115d4895d" containerName="main"
Sep 17 19:20:01.619228 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.619228    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="62c935a0-e5c3-42f8-abd7-3b6115d4895d" containerName="main"
Sep 17 19:20:01.619607 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.619307    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="aef7a70c-cafb-48cd-9ecb-9a729369fb90" containerName="llm-d-routing-sidecar"
Sep 17 19:20:01.619607 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.619321    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="62c935a0-e5c3-42f8-abd7-3b6115d4895d" containerName="main"
Sep 17 19:20:01.619607 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.619332    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="aef7a70c-cafb-48cd-9ecb-9a729369fb90" containerName="main"
Sep 17 19:20:01.623984 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.623921    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:01.626414 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.626389    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-zstlq\""
Sep 17 19:20:01.626567 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.626425    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 17 19:20:01.626567 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.626483    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Sep 17 19:20:01.626567 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.626432    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 17 19:20:01.640395 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.639986    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"]
Sep 17 19:20:01.660355 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.660329    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:01.728443 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.728406    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43983e9f-061a-437b-851d-618c9654116c-istio-token\") pod \"router-gateway-2-openshift-default-844b774c6-zgpkr\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:01.728605 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.728477    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43983e9f-061a-437b-851d-618c9654116c-workload-certs\") pod \"router-gateway-2-openshift-default-844b774c6-zgpkr\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:01.728605 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.728502    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43983e9f-061a-437b-851d-618c9654116c-istio-envoy\") pod \"router-gateway-2-openshift-default-844b774c6-zgpkr\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:01.728605 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.728519    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43983e9f-061a-437b-851d-618c9654116c-istio-podinfo\") pod \"router-gateway-2-openshift-default-844b774c6-zgpkr\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:01.728605 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.728570    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj4qj\" (UniqueName: \"kubernetes.io/projected/43983e9f-061a-437b-851d-618c9654116c-kube-api-access-hj4qj\") pod \"router-gateway-2-openshift-default-844b774c6-zgpkr\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:01.728605 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.728590    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43983e9f-061a-437b-851d-618c9654116c-credential-socket\") pod \"router-gateway-2-openshift-default-844b774c6-zgpkr\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:01.728818 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.728614    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43983e9f-061a-437b-851d-618c9654116c-istio-data\") pod \"router-gateway-2-openshift-default-844b774c6-zgpkr\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:01.728818 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.728702    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43983e9f-061a-437b-851d-618c9654116c-workload-socket\") pod \"router-gateway-2-openshift-default-844b774c6-zgpkr\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:01.728818 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.728732    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43983e9f-061a-437b-851d-618c9654116c-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-844b774c6-zgpkr\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:01.757262 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.757232    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:01.829887 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.829849    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43983e9f-061a-437b-851d-618c9654116c-workload-certs\") pod \"router-gateway-2-openshift-default-844b774c6-zgpkr\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:01.829887 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.829889    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43983e9f-061a-437b-851d-618c9654116c-istio-envoy\") pod \"router-gateway-2-openshift-default-844b774c6-zgpkr\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:01.830165 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.829910    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43983e9f-061a-437b-851d-618c9654116c-istio-podinfo\") pod \"router-gateway-2-openshift-default-844b774c6-zgpkr\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:01.830165 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.829927    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj4qj\" (UniqueName: \"kubernetes.io/projected/43983e9f-061a-437b-851d-618c9654116c-kube-api-access-hj4qj\") pod \"router-gateway-2-openshift-default-844b774c6-zgpkr\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:01.830165 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.830056    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43983e9f-061a-437b-851d-618c9654116c-credential-socket\") pod \"router-gateway-2-openshift-default-844b774c6-zgpkr\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:01.830165 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.830117    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43983e9f-061a-437b-851d-618c9654116c-istio-data\") pod \"router-gateway-2-openshift-default-844b774c6-zgpkr\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:01.830376 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.830176    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43983e9f-061a-437b-851d-618c9654116c-workload-socket\") pod \"router-gateway-2-openshift-default-844b774c6-zgpkr\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:01.830376 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.830246    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43983e9f-061a-437b-851d-618c9654116c-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-844b774c6-zgpkr\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:01.830376 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.830290    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43983e9f-061a-437b-851d-618c9654116c-istio-token\") pod \"router-gateway-2-openshift-default-844b774c6-zgpkr\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:01.830376 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.830357    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43983e9f-061a-437b-851d-618c9654116c-workload-certs\") pod \"router-gateway-2-openshift-default-844b774c6-zgpkr\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:01.830635 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.830397    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43983e9f-061a-437b-851d-618c9654116c-credential-socket\") pod \"router-gateway-2-openshift-default-844b774c6-zgpkr\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:01.830635 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.830616    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43983e9f-061a-437b-851d-618c9654116c-istio-data\") pod \"router-gateway-2-openshift-default-844b774c6-zgpkr\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:01.830727 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.830624    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43983e9f-061a-437b-851d-618c9654116c-workload-socket\") pod \"router-gateway-2-openshift-default-844b774c6-zgpkr\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:01.831077 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.831054    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43983e9f-061a-437b-851d-618c9654116c-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-844b774c6-zgpkr\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:01.832401 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.832380    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43983e9f-061a-437b-851d-618c9654116c-istio-envoy\") pod \"router-gateway-2-openshift-default-844b774c6-zgpkr\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:01.832599 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.832584    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43983e9f-061a-437b-851d-618c9654116c-istio-podinfo\") pod \"router-gateway-2-openshift-default-844b774c6-zgpkr\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:01.837988 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.837969    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43983e9f-061a-437b-851d-618c9654116c-istio-token\") pod \"router-gateway-2-openshift-default-844b774c6-zgpkr\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:01.838094 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.838050    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj4qj\" (UniqueName: \"kubernetes.io/projected/43983e9f-061a-437b-851d-618c9654116c-kube-api-access-hj4qj\") pod \"router-gateway-2-openshift-default-844b774c6-zgpkr\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:01.933896 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:01.933807    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:02.213369 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:02.213314    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"]
Sep 17 19:20:02.215809 ip-10-0-136-132 kubenswrapper[2588]: W0917 19:20:02.215779    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43983e9f_061a_437b_851d_618c9654116c.slice/crio-2c9a5dcb1a0429eff2f4ec7d905b1265cfee19938bf3ff6f3dfa9203c224dd11 WatchSource:0}: Error finding container 2c9a5dcb1a0429eff2f4ec7d905b1265cfee19938bf3ff6f3dfa9203c224dd11: Status 404 returned error can't find the container with id 2c9a5dcb1a0429eff2f4ec7d905b1265cfee19938bf3ff6f3dfa9203c224dd11
Sep 17 19:20:02.218085 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:02.218046    2588 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 19:20:02.218164 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:02.218120    2588 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 19:20:02.218164 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:02.218147    2588 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 19:20:02.460804 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:02.460768    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr" event={"ID":"43983e9f-061a-437b-851d-618c9654116c","Type":"ContainerStarted","Data":"b2d32225fc629275c8ba6012803199c0b2646641305ba975ceed9f08113cdb6d"}
Sep 17 19:20:02.461304 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:02.460819    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr" event={"ID":"43983e9f-061a-437b-851d-618c9654116c","Type":"ContainerStarted","Data":"2c9a5dcb1a0429eff2f4ec7d905b1265cfee19938bf3ff6f3dfa9203c224dd11"}
Sep 17 19:20:02.485016 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:02.484904    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr" podStartSLOduration=1.48488761 podStartE2EDuration="1.48488761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:20:02.483334994 +0000 UTC m=+3039.334340396" watchObservedRunningTime="2026-09-17 19:20:02.48488761 +0000 UTC m=+3039.335893009"
Sep 17 19:20:02.934815 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:02.934774    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:03.958062 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:03.958017    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:04.469266 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:04.469227    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:04.470167 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:04.470147    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:04.657842 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:04.657809    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:07.982246 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:07.982137    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:08.634616 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.634583    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"]
Sep 17 19:20:08.638269 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.638241    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:20:08.640874 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.640849    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:20:08.641106 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.641080    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 17 19:20:08.641277 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.641225    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-2c4zw\""
Sep 17 19:20:08.641363 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.641278    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 17 19:20:08.641442 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.641385    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-hj8w4\""
Sep 17 19:20:08.666496 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.666470    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"]
Sep 17 19:20:08.731710 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.731675    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:08.795835 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.795790    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/110d05aa-592c-42a0-8352-cc5e53a773b2-dshm\") pod \"router-with-refs-pd-test-kserve-648495b7fc-ldjrf\" (UID: \"110d05aa-592c-42a0-8352-cc5e53a773b2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:20:08.795835 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.795839    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/110d05aa-592c-42a0-8352-cc5e53a773b2-home\") pod \"router-with-refs-pd-test-kserve-648495b7fc-ldjrf\" (UID: \"110d05aa-592c-42a0-8352-cc5e53a773b2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:20:08.796103 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.795901    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/110d05aa-592c-42a0-8352-cc5e53a773b2-model-cache\") pod \"router-with-refs-pd-test-kserve-648495b7fc-ldjrf\" (UID: \"110d05aa-592c-42a0-8352-cc5e53a773b2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:20:08.796103 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.795936    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/110d05aa-592c-42a0-8352-cc5e53a773b2-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-648495b7fc-ldjrf\" (UID: \"110d05aa-592c-42a0-8352-cc5e53a773b2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:20:08.796103 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.795958    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/110d05aa-592c-42a0-8352-cc5e53a773b2-tls-certs\") pod \"router-with-refs-pd-test-kserve-648495b7fc-ldjrf\" (UID: \"110d05aa-592c-42a0-8352-cc5e53a773b2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:20:08.796103 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.796002    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/110d05aa-592c-42a0-8352-cc5e53a773b2-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-648495b7fc-ldjrf\" (UID: \"110d05aa-592c-42a0-8352-cc5e53a773b2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:20:08.796103 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.796022    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/110d05aa-592c-42a0-8352-cc5e53a773b2-tmp-dir\") pod \"router-with-refs-pd-test-kserve-648495b7fc-ldjrf\" (UID: \"110d05aa-592c-42a0-8352-cc5e53a773b2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:20:08.796103 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.796056    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fpgq\" (UniqueName: \"kubernetes.io/projected/110d05aa-592c-42a0-8352-cc5e53a773b2-kube-api-access-7fpgq\") pod \"router-with-refs-pd-test-kserve-648495b7fc-ldjrf\" (UID: \"110d05aa-592c-42a0-8352-cc5e53a773b2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:20:08.897323 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.897213    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/110d05aa-592c-42a0-8352-cc5e53a773b2-dshm\") pod \"router-with-refs-pd-test-kserve-648495b7fc-ldjrf\" (UID: \"110d05aa-592c-42a0-8352-cc5e53a773b2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:20:08.897323 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.897256    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/110d05aa-592c-42a0-8352-cc5e53a773b2-home\") pod \"router-with-refs-pd-test-kserve-648495b7fc-ldjrf\" (UID: \"110d05aa-592c-42a0-8352-cc5e53a773b2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:20:08.897323 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.897294    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/110d05aa-592c-42a0-8352-cc5e53a773b2-model-cache\") pod \"router-with-refs-pd-test-kserve-648495b7fc-ldjrf\" (UID: \"110d05aa-592c-42a0-8352-cc5e53a773b2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:20:08.897671 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.897327    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/110d05aa-592c-42a0-8352-cc5e53a773b2-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-648495b7fc-ldjrf\" (UID: \"110d05aa-592c-42a0-8352-cc5e53a773b2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:20:08.897671 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.897354    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/110d05aa-592c-42a0-8352-cc5e53a773b2-tls-certs\") pod \"router-with-refs-pd-test-kserve-648495b7fc-ldjrf\" (UID: \"110d05aa-592c-42a0-8352-cc5e53a773b2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:20:08.897671 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.897392    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/110d05aa-592c-42a0-8352-cc5e53a773b2-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-648495b7fc-ldjrf\" (UID: \"110d05aa-592c-42a0-8352-cc5e53a773b2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:20:08.897671 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.897418    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/110d05aa-592c-42a0-8352-cc5e53a773b2-tmp-dir\") pod \"router-with-refs-pd-test-kserve-648495b7fc-ldjrf\" (UID: \"110d05aa-592c-42a0-8352-cc5e53a773b2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:20:08.897671 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.897462    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fpgq\" (UniqueName: \"kubernetes.io/projected/110d05aa-592c-42a0-8352-cc5e53a773b2-kube-api-access-7fpgq\") pod \"router-with-refs-pd-test-kserve-648495b7fc-ldjrf\" (UID: \"110d05aa-592c-42a0-8352-cc5e53a773b2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:20:08.897934 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.897753    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/110d05aa-592c-42a0-8352-cc5e53a773b2-model-cache\") pod \"router-with-refs-pd-test-kserve-648495b7fc-ldjrf\" (UID: \"110d05aa-592c-42a0-8352-cc5e53a773b2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:20:08.897934 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.897757    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/110d05aa-592c-42a0-8352-cc5e53a773b2-home\") pod \"router-with-refs-pd-test-kserve-648495b7fc-ldjrf\" (UID: \"110d05aa-592c-42a0-8352-cc5e53a773b2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:20:08.898027 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.898002    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/110d05aa-592c-42a0-8352-cc5e53a773b2-tmp-dir\") pod \"router-with-refs-pd-test-kserve-648495b7fc-ldjrf\" (UID: \"110d05aa-592c-42a0-8352-cc5e53a773b2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:20:08.898072 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.898020    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/110d05aa-592c-42a0-8352-cc5e53a773b2-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-648495b7fc-ldjrf\" (UID: \"110d05aa-592c-42a0-8352-cc5e53a773b2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:20:08.898118 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.898107    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/110d05aa-592c-42a0-8352-cc5e53a773b2-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-648495b7fc-ldjrf\" (UID: \"110d05aa-592c-42a0-8352-cc5e53a773b2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:20:08.899610 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.899586    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/110d05aa-592c-42a0-8352-cc5e53a773b2-dshm\") pod \"router-with-refs-pd-test-kserve-648495b7fc-ldjrf\" (UID: \"110d05aa-592c-42a0-8352-cc5e53a773b2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:20:08.900071 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.900025    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/110d05aa-592c-42a0-8352-cc5e53a773b2-tls-certs\") pod \"router-with-refs-pd-test-kserve-648495b7fc-ldjrf\" (UID: \"110d05aa-592c-42a0-8352-cc5e53a773b2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:20:08.908379 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.908350    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fpgq\" (UniqueName: \"kubernetes.io/projected/110d05aa-592c-42a0-8352-cc5e53a773b2-kube-api-access-7fpgq\") pod \"router-with-refs-pd-test-kserve-648495b7fc-ldjrf\" (UID: \"110d05aa-592c-42a0-8352-cc5e53a773b2\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:20:08.948378 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:08.948335    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:20:09.083570 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:09.083511    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv"]
Sep 17 19:20:09.087733 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:09.087715    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv"
Sep 17 19:20:09.090794 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:09.090776    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-tm6l5\""
Sep 17 19:20:09.097057 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:09.097032    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv"]
Sep 17 19:20:09.209616 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:09.199759    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2d2b33-6b1d-4667-b6f6-2988cca1a615-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv\" (UID: \"2d2d2b33-6b1d-4667-b6f6-2988cca1a615\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv"
Sep 17 19:20:09.209616 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:09.199860    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd5kk\" (UniqueName: \"kubernetes.io/projected/2d2d2b33-6b1d-4667-b6f6-2988cca1a615-kube-api-access-kd5kk\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv\" (UID: \"2d2d2b33-6b1d-4667-b6f6-2988cca1a615\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv"
Sep 17 19:20:09.258089 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:09.258055    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"]
Sep 17 19:20:09.260774 ip-10-0-136-132 kubenswrapper[2588]: W0917 19:20:09.260736    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod110d05aa_592c_42a0_8352_cc5e53a773b2.slice/crio-e8ec8c22a76b95738b9f9f0fb1a87ac7539ef71447e2267ea46513ad2be67d5d WatchSource:0}: Error finding container e8ec8c22a76b95738b9f9f0fb1a87ac7539ef71447e2267ea46513ad2be67d5d: Status 404 returned error can't find the container with id e8ec8c22a76b95738b9f9f0fb1a87ac7539ef71447e2267ea46513ad2be67d5d
Sep 17 19:20:09.306562 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:09.304085    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2d2b33-6b1d-4667-b6f6-2988cca1a615-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv\" (UID: \"2d2d2b33-6b1d-4667-b6f6-2988cca1a615\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv"
Sep 17 19:20:09.306562 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:09.304151    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd5kk\" (UniqueName: \"kubernetes.io/projected/2d2d2b33-6b1d-4667-b6f6-2988cca1a615-kube-api-access-kd5kk\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv\" (UID: \"2d2d2b33-6b1d-4667-b6f6-2988cca1a615\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv"
Sep 17 19:20:09.307999 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:09.307956    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2d2b33-6b1d-4667-b6f6-2988cca1a615-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv\" (UID: \"2d2d2b33-6b1d-4667-b6f6-2988cca1a615\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv"
Sep 17 19:20:09.313932 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:09.313889    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:09.316143 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:09.316094    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd5kk\" (UniqueName: \"kubernetes.io/projected/2d2d2b33-6b1d-4667-b6f6-2988cca1a615-kube-api-access-kd5kk\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv\" (UID: \"2d2d2b33-6b1d-4667-b6f6-2988cca1a615\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv"
Sep 17 19:20:09.398370 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:09.398334    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv"
Sep 17 19:20:09.428850 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:09.422239    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:09.492346 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:09.492312    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf" event={"ID":"110d05aa-592c-42a0-8352-cc5e53a773b2","Type":"ContainerStarted","Data":"447f19caf69385c40541c1e790f4d8cdd2b29732bf7ded7682fc019fc9e43e5c"}
Sep 17 19:20:09.492489 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:09.492381    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf" event={"ID":"110d05aa-592c-42a0-8352-cc5e53a773b2","Type":"ContainerStarted","Data":"e8ec8c22a76b95738b9f9f0fb1a87ac7539ef71447e2267ea46513ad2be67d5d"}
Sep 17 19:20:09.492489 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:09.492415    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:20:09.696039 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:09.696006    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv"]
Sep 17 19:20:09.699521 ip-10-0-136-132 kubenswrapper[2588]: W0917 19:20:09.699490    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d2d2b33_6b1d_4667_b6f6_2988cca1a615.slice/crio-70e7da6a289c1519c3eedae0f5c6c1112c9b0ab47e70e269934850d71e6fb275 WatchSource:0}: Error finding container 70e7da6a289c1519c3eedae0f5c6c1112c9b0ab47e70e269934850d71e6fb275: Status 404 returned error can't find the container with id 70e7da6a289c1519c3eedae0f5c6c1112c9b0ab47e70e269934850d71e6fb275
Sep 17 19:20:10.500567 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:10.500371    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv" event={"ID":"2d2d2b33-6b1d-4667-b6f6-2988cca1a615","Type":"ContainerStarted","Data":"2c612bdae22929941a60544e18a6847469c999e2455fe6ae0e787d5e26fe2dd5"}
Sep 17 19:20:10.500567 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:10.500431    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv" event={"ID":"2d2d2b33-6b1d-4667-b6f6-2988cca1a615","Type":"ContainerStarted","Data":"70e7da6a289c1519c3eedae0f5c6c1112c9b0ab47e70e269934850d71e6fb275"}
Sep 17 19:20:10.501589 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:10.501563    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv"
Sep 17 19:20:10.503671 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:10.503631    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf" event={"ID":"110d05aa-592c-42a0-8352-cc5e53a773b2","Type":"ContainerStarted","Data":"64c5ac8e510865dd188cb5e22600d1140693c0ad99d401f3156e475f707228d1"}
Sep 17 19:20:10.548596 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:10.548540    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv" podStartSLOduration=1.5485095 podStartE2EDuration="1.5485095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:20:10.522831457 +0000 UTC m=+3047.373836858" watchObservedRunningTime="2026-09-17 19:20:10.5485095 +0000 UTC m=+3047.399514899"
Sep 17 19:20:12.518146 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:12.518105    2588 generic.go:356] "Generic (PLEG): container finished" podID="110d05aa-592c-42a0-8352-cc5e53a773b2" containerID="64c5ac8e510865dd188cb5e22600d1140693c0ad99d401f3156e475f707228d1" exitCode=0
Sep 17 19:20:12.518676 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:12.518178    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf" event={"ID":"110d05aa-592c-42a0-8352-cc5e53a773b2","Type":"ContainerDied","Data":"64c5ac8e510865dd188cb5e22600d1140693c0ad99d401f3156e475f707228d1"}
Sep 17 19:20:13.525102 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:13.524987    2588 generic.go:356] "Generic (PLEG): container finished" podID="43983e9f-061a-437b-851d-618c9654116c" containerID="b2d32225fc629275c8ba6012803199c0b2646641305ba975ceed9f08113cdb6d" exitCode=0
Sep 17 19:20:13.525102 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:13.525059    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr" event={"ID":"43983e9f-061a-437b-851d-618c9654116c","Type":"ContainerDied","Data":"b2d32225fc629275c8ba6012803199c0b2646641305ba975ceed9f08113cdb6d"}
Sep 17 19:20:13.525668 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:13.525494    2588 scope.go:117] "RemoveContainer" containerID="b2d32225fc629275c8ba6012803199c0b2646641305ba975ceed9f08113cdb6d"
Sep 17 19:20:13.526565 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:13.526516    2588 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 19:20:13.526719 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:13.526621    2588 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 19:20:13.526719 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:13.526675    2588 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 19:20:13.527572 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:13.527548    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf" event={"ID":"110d05aa-592c-42a0-8352-cc5e53a773b2","Type":"ContainerStarted","Data":"c9d795ec740018dcb05eb1db1047a9db855fa89bcba2389c3b9c37e39c6f581d"}
Sep 17 19:20:13.574185 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:13.573616    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf" podStartSLOduration=3.164281602 podStartE2EDuration="5.573592865s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.409311263s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:20:13.570994129 +0000 UTC m=+3050.421999564" watchObservedRunningTime="2026-09-17 19:20:13.573592865 +0000 UTC m=+3050.424598266"
Sep 17 19:20:13.715380 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:13.715337    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:13.790654 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:13.790515    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:14.534311 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:14.534276    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr" event={"ID":"43983e9f-061a-437b-851d-618c9654116c","Type":"ContainerStarted","Data":"516037291d71a743b329ad54c67c07c508ba9346881eba90bf9f83070b31909a"}
Sep 17 19:20:14.934853 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:14.934809    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:14.936153 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:14.936124    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr" podUID="43983e9f-061a-437b-851d-618c9654116c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 17 19:20:15.934493 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:15.934453    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr" podUID="43983e9f-061a-437b-851d-618c9654116c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 17 19:20:16.934372 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:16.934329    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr" podUID="43983e9f-061a-437b-851d-618c9654116c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 17 19:20:17.934190 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:17.934148    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr" podUID="43983e9f-061a-437b-851d-618c9654116c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 17 19:20:18.934734 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:18.934687    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr" podUID="43983e9f-061a-437b-851d-618c9654116c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 17 19:20:18.949465 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:18.949434    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:20:18.949640 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:18.949593    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:20:18.950797 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:18.950763    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf" podUID="110d05aa-592c-42a0-8352-cc5e53a773b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Sep 17 19:20:19.470142 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:19.470103    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:19.568237 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:19.568205    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:20:19.938616 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:19.938584    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:19.939691 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:19.939669    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:20:28.949005 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:28.948954    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf" podUID="110d05aa-592c-42a0-8352-cc5e53a773b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Sep 17 19:20:37.629699 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:37.629665    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:37.694302 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:37.694272    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:38.949157 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:38.949114    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf" podUID="110d05aa-592c-42a0-8352-cc5e53a773b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Sep 17 19:20:41.513691 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:41.513662    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv"
Sep 17 19:20:48.949393 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:48.949332    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf" podUID="110d05aa-592c-42a0-8352-cc5e53a773b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Sep 17 19:20:52.863235 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:52.863200    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:58.948909 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:20:58.948850    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf" podUID="110d05aa-592c-42a0-8352-cc5e53a773b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Sep 17 19:21:08.949647 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:21:08.949601    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf" podUID="110d05aa-592c-42a0-8352-cc5e53a773b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Sep 17 19:21:12.913924 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:21:12.913892    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:21:13.014814 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:21:13.014783    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:21:13.113447 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:21:13.113410    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:21:13.215229 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:21:13.215141    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:21:13.340393 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:21:13.340359    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:21:13.438668 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:21:13.438631    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:21:16.190459 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:21:16.190402    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:21:16.290852 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:21:16.290815    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:21:16.437265 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:21:16.437233    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:21:18.760795 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:21:18.760750    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:21:18.861745 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:21:18.861713    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:21:18.949730 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:21:18.949686    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf" podUID="110d05aa-592c-42a0-8352-cc5e53a773b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Sep 17 19:21:28.948967 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:21:28.948924    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf" podUID="110d05aa-592c-42a0-8352-cc5e53a773b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Sep 17 19:21:36.918655 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:21:36.918503    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:21:38.949032 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:21:38.948966    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf" podUID="110d05aa-592c-42a0-8352-cc5e53a773b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Sep 17 19:21:48.949372 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:21:48.949320    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf" podUID="110d05aa-592c-42a0-8352-cc5e53a773b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Sep 17 19:21:51.112977 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:21:51.112942    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:21:58.949156 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:21:58.949110    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf" podUID="110d05aa-592c-42a0-8352-cc5e53a773b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Sep 17 19:22:08.958730 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:08.958696    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:22:08.970674 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:08.970643    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:22:18.253685 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:18.253653    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:22:18.283706 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:18.283675    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"]
Sep 17 19:22:18.286836 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:18.286810    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv"]
Sep 17 19:22:18.333244 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:18.333190    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:22:21.514011 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:21.513967    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv" podUID="2d2d2b33-6b1d-4667-b6f6-2988cca1a615" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:22:31.513625 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:31.513586    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv" podUID="2d2d2b33-6b1d-4667-b6f6-2988cca1a615" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:22:33.305401 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:33.305357    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv" podUID="2d2d2b33-6b1d-4667-b6f6-2988cca1a615" containerName="main" containerID="cri-o://2c612bdae22929941a60544e18a6847469c999e2455fe6ae0e787d5e26fe2dd5" gracePeriod=45
Sep 17 19:22:33.306789 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:33.306754    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf" podUID="110d05aa-592c-42a0-8352-cc5e53a773b2" containerName="main" containerID="cri-o://c9d795ec740018dcb05eb1db1047a9db855fa89bcba2389c3b9c37e39c6f581d" gracePeriod=45
Sep 17 19:22:33.925828 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:33.925767    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf" podUID="110d05aa-592c-42a0-8352-cc5e53a773b2" containerName="llm-d-routing-sidecar" containerID="cri-o://447f19caf69385c40541c1e790f4d8cdd2b29732bf7ded7682fc019fc9e43e5c" gracePeriod=45
Sep 17 19:22:34.162723 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.162694    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:22:34.188411 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.188382    2588 generic.go:356] "Generic (PLEG): container finished" podID="110d05aa-592c-42a0-8352-cc5e53a773b2" containerID="c9d795ec740018dcb05eb1db1047a9db855fa89bcba2389c3b9c37e39c6f581d" exitCode=0
Sep 17 19:22:34.188411 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.188405    2588 generic.go:356] "Generic (PLEG): container finished" podID="110d05aa-592c-42a0-8352-cc5e53a773b2" containerID="447f19caf69385c40541c1e790f4d8cdd2b29732bf7ded7682fc019fc9e43e5c" exitCode=0
Sep 17 19:22:34.188623 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.188437    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"
Sep 17 19:22:34.188623 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.188460    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf" event={"ID":"110d05aa-592c-42a0-8352-cc5e53a773b2","Type":"ContainerDied","Data":"c9d795ec740018dcb05eb1db1047a9db855fa89bcba2389c3b9c37e39c6f581d"}
Sep 17 19:22:34.188623 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.188517    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf" event={"ID":"110d05aa-592c-42a0-8352-cc5e53a773b2","Type":"ContainerDied","Data":"447f19caf69385c40541c1e790f4d8cdd2b29732bf7ded7682fc019fc9e43e5c"}
Sep 17 19:22:34.188623 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.188557    2588 scope.go:117] "RemoveContainer" containerID="c9d795ec740018dcb05eb1db1047a9db855fa89bcba2389c3b9c37e39c6f581d"
Sep 17 19:22:34.188623 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.188560    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf" event={"ID":"110d05aa-592c-42a0-8352-cc5e53a773b2","Type":"ContainerDied","Data":"e8ec8c22a76b95738b9f9f0fb1a87ac7539ef71447e2267ea46513ad2be67d5d"}
Sep 17 19:22:34.197918 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.197896    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/110d05aa-592c-42a0-8352-cc5e53a773b2-dshm\") pod \"110d05aa-592c-42a0-8352-cc5e53a773b2\" (UID: \"110d05aa-592c-42a0-8352-cc5e53a773b2\") "
Sep 17 19:22:34.198064 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.197937    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/110d05aa-592c-42a0-8352-cc5e53a773b2-tls-certs\") pod \"110d05aa-592c-42a0-8352-cc5e53a773b2\" (UID: \"110d05aa-592c-42a0-8352-cc5e53a773b2\") "
Sep 17 19:22:34.198064 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.197982    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/110d05aa-592c-42a0-8352-cc5e53a773b2-home\") pod \"110d05aa-592c-42a0-8352-cc5e53a773b2\" (UID: \"110d05aa-592c-42a0-8352-cc5e53a773b2\") "
Sep 17 19:22:34.198064 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.198006    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/110d05aa-592c-42a0-8352-cc5e53a773b2-cabundle-cert\") pod \"110d05aa-592c-42a0-8352-cc5e53a773b2\" (UID: \"110d05aa-592c-42a0-8352-cc5e53a773b2\") "
Sep 17 19:22:34.198477 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.198376    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/110d05aa-592c-42a0-8352-cc5e53a773b2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "110d05aa-592c-42a0-8352-cc5e53a773b2" (UID: "110d05aa-592c-42a0-8352-cc5e53a773b2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:22:34.198477 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.198471    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/110d05aa-592c-42a0-8352-cc5e53a773b2-model-cache\") pod \"110d05aa-592c-42a0-8352-cc5e53a773b2\" (UID: \"110d05aa-592c-42a0-8352-cc5e53a773b2\") "
Sep 17 19:22:34.198681 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.198558    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/110d05aa-592c-42a0-8352-cc5e53a773b2-tmp-dir\") pod \"110d05aa-592c-42a0-8352-cc5e53a773b2\" (UID: \"110d05aa-592c-42a0-8352-cc5e53a773b2\") "
Sep 17 19:22:34.198681 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.198592    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/110d05aa-592c-42a0-8352-cc5e53a773b2-kserve-provision-location\") pod \"110d05aa-592c-42a0-8352-cc5e53a773b2\" (UID: \"110d05aa-592c-42a0-8352-cc5e53a773b2\") "
Sep 17 19:22:34.198681 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.198630    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fpgq\" (UniqueName: \"kubernetes.io/projected/110d05aa-592c-42a0-8352-cc5e53a773b2-kube-api-access-7fpgq\") pod \"110d05aa-592c-42a0-8352-cc5e53a773b2\" (UID: \"110d05aa-592c-42a0-8352-cc5e53a773b2\") "
Sep 17 19:22:34.198843 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.198761    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/110d05aa-592c-42a0-8352-cc5e53a773b2-model-cache" (OuterVolumeSpecName: "model-cache") pod "110d05aa-592c-42a0-8352-cc5e53a773b2" (UID: "110d05aa-592c-42a0-8352-cc5e53a773b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:22:34.199077 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.198960    2588 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/110d05aa-592c-42a0-8352-cc5e53a773b2-cabundle-cert\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:22:34.199077 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.198981    2588 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/110d05aa-592c-42a0-8352-cc5e53a773b2-model-cache\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:22:34.199191 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.199151    2588 scope.go:117] "RemoveContainer" containerID="64c5ac8e510865dd188cb5e22600d1140693c0ad99d401f3156e475f707228d1"
Sep 17 19:22:34.199705 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.199667    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/110d05aa-592c-42a0-8352-cc5e53a773b2-home" (OuterVolumeSpecName: "home") pod "110d05aa-592c-42a0-8352-cc5e53a773b2" (UID: "110d05aa-592c-42a0-8352-cc5e53a773b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:22:34.200900 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.200849    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/110d05aa-592c-42a0-8352-cc5e53a773b2-dshm" (OuterVolumeSpecName: "dshm") pod "110d05aa-592c-42a0-8352-cc5e53a773b2" (UID: "110d05aa-592c-42a0-8352-cc5e53a773b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:22:34.201175 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.201138    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/110d05aa-592c-42a0-8352-cc5e53a773b2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "110d05aa-592c-42a0-8352-cc5e53a773b2" (UID: "110d05aa-592c-42a0-8352-cc5e53a773b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:22:34.202835 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.202807    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/110d05aa-592c-42a0-8352-cc5e53a773b2-kube-api-access-7fpgq" (OuterVolumeSpecName: "kube-api-access-7fpgq") pod "110d05aa-592c-42a0-8352-cc5e53a773b2" (UID: "110d05aa-592c-42a0-8352-cc5e53a773b2"). InnerVolumeSpecName "kube-api-access-7fpgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:22:34.217742 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.217715    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/110d05aa-592c-42a0-8352-cc5e53a773b2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "110d05aa-592c-42a0-8352-cc5e53a773b2" (UID: "110d05aa-592c-42a0-8352-cc5e53a773b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:22:34.217862 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.217825    2588 scope.go:117] "RemoveContainer" containerID="447f19caf69385c40541c1e790f4d8cdd2b29732bf7ded7682fc019fc9e43e5c"
Sep 17 19:22:34.226614 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.226595    2588 scope.go:117] "RemoveContainer" containerID="c9d795ec740018dcb05eb1db1047a9db855fa89bcba2389c3b9c37e39c6f581d"
Sep 17 19:22:34.226922 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:22:34.226901    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9d795ec740018dcb05eb1db1047a9db855fa89bcba2389c3b9c37e39c6f581d\": container with ID starting with c9d795ec740018dcb05eb1db1047a9db855fa89bcba2389c3b9c37e39c6f581d not found: ID does not exist" containerID="c9d795ec740018dcb05eb1db1047a9db855fa89bcba2389c3b9c37e39c6f581d"
Sep 17 19:22:34.226979 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.226927    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/110d05aa-592c-42a0-8352-cc5e53a773b2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "110d05aa-592c-42a0-8352-cc5e53a773b2" (UID: "110d05aa-592c-42a0-8352-cc5e53a773b2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:22:34.226979 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.226929    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9d795ec740018dcb05eb1db1047a9db855fa89bcba2389c3b9c37e39c6f581d"} err="failed to get container status \"c9d795ec740018dcb05eb1db1047a9db855fa89bcba2389c3b9c37e39c6f581d\": rpc error: code = NotFound desc = could not find container \"c9d795ec740018dcb05eb1db1047a9db855fa89bcba2389c3b9c37e39c6f581d\": container with ID starting with c9d795ec740018dcb05eb1db1047a9db855fa89bcba2389c3b9c37e39c6f581d not found: ID does not exist"
Sep 17 19:22:34.226979 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.226971    2588 scope.go:117] "RemoveContainer" containerID="64c5ac8e510865dd188cb5e22600d1140693c0ad99d401f3156e475f707228d1"
Sep 17 19:22:34.227255 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:22:34.227238    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64c5ac8e510865dd188cb5e22600d1140693c0ad99d401f3156e475f707228d1\": container with ID starting with 64c5ac8e510865dd188cb5e22600d1140693c0ad99d401f3156e475f707228d1 not found: ID does not exist" containerID="64c5ac8e510865dd188cb5e22600d1140693c0ad99d401f3156e475f707228d1"
Sep 17 19:22:34.227309 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.227259    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64c5ac8e510865dd188cb5e22600d1140693c0ad99d401f3156e475f707228d1"} err="failed to get container status \"64c5ac8e510865dd188cb5e22600d1140693c0ad99d401f3156e475f707228d1\": rpc error: code = NotFound desc = could not find container \"64c5ac8e510865dd188cb5e22600d1140693c0ad99d401f3156e475f707228d1\": container with ID starting with 64c5ac8e510865dd188cb5e22600d1140693c0ad99d401f3156e475f707228d1 not found: ID does not exist"
Sep 17 19:22:34.227309 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.227275    2588 scope.go:117] "RemoveContainer" containerID="447f19caf69385c40541c1e790f4d8cdd2b29732bf7ded7682fc019fc9e43e5c"
Sep 17 19:22:34.227492 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:22:34.227472    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"447f19caf69385c40541c1e790f4d8cdd2b29732bf7ded7682fc019fc9e43e5c\": container with ID starting with 447f19caf69385c40541c1e790f4d8cdd2b29732bf7ded7682fc019fc9e43e5c not found: ID does not exist" containerID="447f19caf69385c40541c1e790f4d8cdd2b29732bf7ded7682fc019fc9e43e5c"
Sep 17 19:22:34.227738 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.227499    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"447f19caf69385c40541c1e790f4d8cdd2b29732bf7ded7682fc019fc9e43e5c"} err="failed to get container status \"447f19caf69385c40541c1e790f4d8cdd2b29732bf7ded7682fc019fc9e43e5c\": rpc error: code = NotFound desc = could not find container \"447f19caf69385c40541c1e790f4d8cdd2b29732bf7ded7682fc019fc9e43e5c\": container with ID starting with 447f19caf69385c40541c1e790f4d8cdd2b29732bf7ded7682fc019fc9e43e5c not found: ID does not exist"
Sep 17 19:22:34.227738 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.227516    2588 scope.go:117] "RemoveContainer" containerID="c9d795ec740018dcb05eb1db1047a9db855fa89bcba2389c3b9c37e39c6f581d"
Sep 17 19:22:34.227841 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.227770    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9d795ec740018dcb05eb1db1047a9db855fa89bcba2389c3b9c37e39c6f581d"} err="failed to get container status \"c9d795ec740018dcb05eb1db1047a9db855fa89bcba2389c3b9c37e39c6f581d\": rpc error: code = NotFound desc = could not find container \"c9d795ec740018dcb05eb1db1047a9db855fa89bcba2389c3b9c37e39c6f581d\": container with ID starting with c9d795ec740018dcb05eb1db1047a9db855fa89bcba2389c3b9c37e39c6f581d not found: ID does not exist"
Sep 17 19:22:34.227841 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.227789    2588 scope.go:117] "RemoveContainer" containerID="64c5ac8e510865dd188cb5e22600d1140693c0ad99d401f3156e475f707228d1"
Sep 17 19:22:34.228024 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.228006    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64c5ac8e510865dd188cb5e22600d1140693c0ad99d401f3156e475f707228d1"} err="failed to get container status \"64c5ac8e510865dd188cb5e22600d1140693c0ad99d401f3156e475f707228d1\": rpc error: code = NotFound desc = could not find container \"64c5ac8e510865dd188cb5e22600d1140693c0ad99d401f3156e475f707228d1\": container with ID starting with 64c5ac8e510865dd188cb5e22600d1140693c0ad99d401f3156e475f707228d1 not found: ID does not exist"
Sep 17 19:22:34.228068 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.228024    2588 scope.go:117] "RemoveContainer" containerID="447f19caf69385c40541c1e790f4d8cdd2b29732bf7ded7682fc019fc9e43e5c"
Sep 17 19:22:34.228295 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.228270    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"447f19caf69385c40541c1e790f4d8cdd2b29732bf7ded7682fc019fc9e43e5c"} err="failed to get container status \"447f19caf69385c40541c1e790f4d8cdd2b29732bf7ded7682fc019fc9e43e5c\": rpc error: code = NotFound desc = could not find container \"447f19caf69385c40541c1e790f4d8cdd2b29732bf7ded7682fc019fc9e43e5c\": container with ID starting with 447f19caf69385c40541c1e790f4d8cdd2b29732bf7ded7682fc019fc9e43e5c not found: ID does not exist"
Sep 17 19:22:34.299879 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.299839    2588 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/110d05aa-592c-42a0-8352-cc5e53a773b2-tmp-dir\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:22:34.299879 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.299875    2588 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/110d05aa-592c-42a0-8352-cc5e53a773b2-kserve-provision-location\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:22:34.299879 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.299885    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fpgq\" (UniqueName: \"kubernetes.io/projected/110d05aa-592c-42a0-8352-cc5e53a773b2-kube-api-access-7fpgq\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:22:34.299879 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.299894    2588 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/110d05aa-592c-42a0-8352-cc5e53a773b2-dshm\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:22:34.300099 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.299903    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/110d05aa-592c-42a0-8352-cc5e53a773b2-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:22:34.300099 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.299912    2588 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/110d05aa-592c-42a0-8352-cc5e53a773b2-home\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:22:34.512968 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.512939    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"]
Sep 17 19:22:34.516541 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:34.516506    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-648495b7fc-ldjrf"]
Sep 17 19:22:35.505824 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:35.505777    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="110d05aa-592c-42a0-8352-cc5e53a773b2" path="/var/lib/kubelet/pods/110d05aa-592c-42a0-8352-cc5e53a773b2/volumes"
Sep 17 19:22:41.513749 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:41.513711    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv" podUID="2d2d2b33-6b1d-4667-b6f6-2988cca1a615" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:22:41.514151 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:41.513821    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv"
Sep 17 19:22:51.513645 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:22:51.513607    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv" podUID="2d2d2b33-6b1d-4667-b6f6-2988cca1a615" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:23:01.514125 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:01.514087    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv" podUID="2d2d2b33-6b1d-4667-b6f6-2988cca1a615" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:23:03.551919 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:03.551883    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv"
Sep 17 19:23:03.653548 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:03.653437    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2d2b33-6b1d-4667-b6f6-2988cca1a615-tls-certs\") pod \"2d2d2b33-6b1d-4667-b6f6-2988cca1a615\" (UID: \"2d2d2b33-6b1d-4667-b6f6-2988cca1a615\") "
Sep 17 19:23:03.653548 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:03.653488    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kd5kk\" (UniqueName: \"kubernetes.io/projected/2d2d2b33-6b1d-4667-b6f6-2988cca1a615-kube-api-access-kd5kk\") pod \"2d2d2b33-6b1d-4667-b6f6-2988cca1a615\" (UID: \"2d2d2b33-6b1d-4667-b6f6-2988cca1a615\") "
Sep 17 19:23:03.655679 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:03.655649    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d2d2b33-6b1d-4667-b6f6-2988cca1a615-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2d2d2b33-6b1d-4667-b6f6-2988cca1a615" (UID: "2d2d2b33-6b1d-4667-b6f6-2988cca1a615"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:23:03.655798 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:03.655702    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2d2b33-6b1d-4667-b6f6-2988cca1a615-kube-api-access-kd5kk" (OuterVolumeSpecName: "kube-api-access-kd5kk") pod "2d2d2b33-6b1d-4667-b6f6-2988cca1a615" (UID: "2d2d2b33-6b1d-4667-b6f6-2988cca1a615"). InnerVolumeSpecName "kube-api-access-kd5kk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:23:03.755119 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:03.755077    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2d2b33-6b1d-4667-b6f6-2988cca1a615-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:23:03.755119 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:03.755111    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd5kk\" (UniqueName: \"kubernetes.io/projected/2d2d2b33-6b1d-4667-b6f6-2988cca1a615-kube-api-access-kd5kk\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:23:04.320108 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:04.320072    2588 generic.go:356] "Generic (PLEG): container finished" podID="2d2d2b33-6b1d-4667-b6f6-2988cca1a615" containerID="2c612bdae22929941a60544e18a6847469c999e2455fe6ae0e787d5e26fe2dd5" exitCode=0
Sep 17 19:23:04.320285 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:04.320113    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv"
Sep 17 19:23:04.320285 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:04.320161    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv" event={"ID":"2d2d2b33-6b1d-4667-b6f6-2988cca1a615","Type":"ContainerDied","Data":"2c612bdae22929941a60544e18a6847469c999e2455fe6ae0e787d5e26fe2dd5"}
Sep 17 19:23:04.320285 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:04.320210    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv" event={"ID":"2d2d2b33-6b1d-4667-b6f6-2988cca1a615","Type":"ContainerDied","Data":"70e7da6a289c1519c3eedae0f5c6c1112c9b0ab47e70e269934850d71e6fb275"}
Sep 17 19:23:04.320285 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:04.320235    2588 scope.go:117] "RemoveContainer" containerID="2c612bdae22929941a60544e18a6847469c999e2455fe6ae0e787d5e26fe2dd5"
Sep 17 19:23:04.328909 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:04.328890    2588 scope.go:117] "RemoveContainer" containerID="2c612bdae22929941a60544e18a6847469c999e2455fe6ae0e787d5e26fe2dd5"
Sep 17 19:23:04.329181 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:23:04.329165    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c612bdae22929941a60544e18a6847469c999e2455fe6ae0e787d5e26fe2dd5\": container with ID starting with 2c612bdae22929941a60544e18a6847469c999e2455fe6ae0e787d5e26fe2dd5 not found: ID does not exist" containerID="2c612bdae22929941a60544e18a6847469c999e2455fe6ae0e787d5e26fe2dd5"
Sep 17 19:23:04.329255 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:04.329186    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c612bdae22929941a60544e18a6847469c999e2455fe6ae0e787d5e26fe2dd5"} err="failed to get container status \"2c612bdae22929941a60544e18a6847469c999e2455fe6ae0e787d5e26fe2dd5\": rpc error: code = NotFound desc = could not find container \"2c612bdae22929941a60544e18a6847469c999e2455fe6ae0e787d5e26fe2dd5\": container with ID starting with 2c612bdae22929941a60544e18a6847469c999e2455fe6ae0e787d5e26fe2dd5 not found: ID does not exist"
Sep 17 19:23:04.349018 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:04.348993    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv"]
Sep 17 19:23:04.359170 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:04.359145    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45p98lv"]
Sep 17 19:23:05.505588 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:05.505547    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d2d2b33-6b1d-4667-b6f6-2988cca1a615" path="/var/lib/kubelet/pods/2d2d2b33-6b1d-4667-b6f6-2988cca1a615/volumes"
Sep 17 19:23:08.290291 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:08.290217    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:08.379752 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:08.379723    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:13.237607 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:13.237572    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"]
Sep 17 19:23:13.238049 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:13.237752    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr" podUID="43983e9f-061a-437b-851d-618c9654116c" containerName="istio-proxy" containerID="cri-o://516037291d71a743b329ad54c67c07c508ba9346881eba90bf9f83070b31909a" gracePeriod=30
Sep 17 19:23:13.335719 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:13.335686    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:18.205997 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.205968    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"]
Sep 17 19:23:18.206368 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.206233    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="110d05aa-592c-42a0-8352-cc5e53a773b2" containerName="llm-d-routing-sidecar"
Sep 17 19:23:18.206368 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.206244    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="110d05aa-592c-42a0-8352-cc5e53a773b2" containerName="llm-d-routing-sidecar"
Sep 17 19:23:18.206368 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.206254    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="110d05aa-592c-42a0-8352-cc5e53a773b2" containerName="storage-initializer"
Sep 17 19:23:18.206368 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.206260    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="110d05aa-592c-42a0-8352-cc5e53a773b2" containerName="storage-initializer"
Sep 17 19:23:18.206368 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.206279    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d2d2b33-6b1d-4667-b6f6-2988cca1a615" containerName="main"
Sep 17 19:23:18.206368 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.206285    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d2d2b33-6b1d-4667-b6f6-2988cca1a615" containerName="main"
Sep 17 19:23:18.206368 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.206292    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="110d05aa-592c-42a0-8352-cc5e53a773b2" containerName="main"
Sep 17 19:23:18.206368 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.206297    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="110d05aa-592c-42a0-8352-cc5e53a773b2" containerName="main"
Sep 17 19:23:18.206368 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.206344    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d2d2b33-6b1d-4667-b6f6-2988cca1a615" containerName="main"
Sep 17 19:23:18.206368 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.206351    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="110d05aa-592c-42a0-8352-cc5e53a773b2" containerName="main"
Sep 17 19:23:18.206368 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.206359    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="110d05aa-592c-42a0-8352-cc5e53a773b2" containerName="llm-d-routing-sidecar"
Sep 17 19:23:18.210665 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.210647    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"
Sep 17 19:23:18.213321 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.213299    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Sep 17 19:23:18.213431 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.213299    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-2rlp9\""
Sep 17 19:23:18.213431 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.213300    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Sep 17 19:23:18.213431 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.213326    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Sep 17 19:23:18.213658 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.213642    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:23:18.213861 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.213846    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Sep 17 19:23:18.224734 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.224712    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"]
Sep 17 19:23:18.226840 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.226819    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:18.261975 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.261953    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7\" (UID: \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"
Sep 17 19:23:18.262068 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.261980    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7\" (UID: \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"
Sep 17 19:23:18.262068 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.261999    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7\" (UID: \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"
Sep 17 19:23:18.262068 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.262023    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7\" (UID: \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"
Sep 17 19:23:18.262213 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.262126    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7\" (UID: \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"
Sep 17 19:23:18.262213 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.262173    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7\" (UID: \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"
Sep 17 19:23:18.262213 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.262193    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7\" (UID: \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"
Sep 17 19:23:18.262331 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.262219    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7fr9\" (UniqueName: \"kubernetes.io/projected/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-kube-api-access-m7fr9\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7\" (UID: \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"
Sep 17 19:23:18.363166 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.363142    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7\" (UID: \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"
Sep 17 19:23:18.363297 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.363209    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7\" (UID: \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"
Sep 17 19:23:18.363297 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.363245    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7\" (UID: \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"
Sep 17 19:23:18.363297 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.363272    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7\" (UID: \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"
Sep 17 19:23:18.363471 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.363307    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fr9\" (UniqueName: \"kubernetes.io/projected/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-kube-api-access-m7fr9\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7\" (UID: \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"
Sep 17 19:23:18.363471 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.363359    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7\" (UID: \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"
Sep 17 19:23:18.363471 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.363386    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7\" (UID: \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"
Sep 17 19:23:18.363471 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.363412    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7\" (UID: \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"
Sep 17 19:23:18.363688 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.363521    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7\" (UID: \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"
Sep 17 19:23:18.363688 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.363586    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7\" (UID: \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"
Sep 17 19:23:18.363898 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.363871    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7\" (UID: \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"
Sep 17 19:23:18.364024 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.363967    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7\" (UID: \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"
Sep 17 19:23:18.364024 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.363979    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7\" (UID: \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"
Sep 17 19:23:18.366130 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.366104    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7\" (UID: \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"
Sep 17 19:23:18.366397 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.366380    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7\" (UID: \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"
Sep 17 19:23:18.371644 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.371623    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7fr9\" (UniqueName: \"kubernetes.io/projected/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-kube-api-access-m7fr9\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7\" (UID: \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"
Sep 17 19:23:18.380907 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.380886    2588 generic.go:356] "Generic (PLEG): container finished" podID="43983e9f-061a-437b-851d-618c9654116c" containerID="516037291d71a743b329ad54c67c07c508ba9346881eba90bf9f83070b31909a" exitCode=0
Sep 17 19:23:18.380991 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.380962    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr" event={"ID":"43983e9f-061a-437b-851d-618c9654116c","Type":"ContainerDied","Data":"516037291d71a743b329ad54c67c07c508ba9346881eba90bf9f83070b31909a"}
Sep 17 19:23:18.381041 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.381018    2588 scope.go:117] "RemoveContainer" containerID="b2d32225fc629275c8ba6012803199c0b2646641305ba975ceed9f08113cdb6d"
Sep 17 19:23:18.490641 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.490615    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:23:18.520868 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.520807    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"
Sep 17 19:23:18.565281 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.565246    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43983e9f-061a-437b-851d-618c9654116c-workload-socket\") pod \"43983e9f-061a-437b-851d-618c9654116c\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") "
Sep 17 19:23:18.565558 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.565492    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43983e9f-061a-437b-851d-618c9654116c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "43983e9f-061a-437b-851d-618c9654116c" (UID: "43983e9f-061a-437b-851d-618c9654116c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:23:18.565644 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.565629    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43983e9f-061a-437b-851d-618c9654116c-istiod-ca-cert\") pod \"43983e9f-061a-437b-851d-618c9654116c\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") "
Sep 17 19:23:18.566009 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.565986    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43983e9f-061a-437b-851d-618c9654116c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "43983e9f-061a-437b-851d-618c9654116c" (UID: "43983e9f-061a-437b-851d-618c9654116c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:23:18.566066 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.566051    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43983e9f-061a-437b-851d-618c9654116c-istio-token\") pod \"43983e9f-061a-437b-851d-618c9654116c\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") "
Sep 17 19:23:18.566330 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.566307    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43983e9f-061a-437b-851d-618c9654116c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "43983e9f-061a-437b-851d-618c9654116c" (UID: "43983e9f-061a-437b-851d-618c9654116c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:23:18.566548 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.566512    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43983e9f-061a-437b-851d-618c9654116c-workload-certs\") pod \"43983e9f-061a-437b-851d-618c9654116c\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") "
Sep 17 19:23:18.566653 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.566604    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43983e9f-061a-437b-851d-618c9654116c-credential-socket\") pod \"43983e9f-061a-437b-851d-618c9654116c\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") "
Sep 17 19:23:18.566698 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.566654    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43983e9f-061a-437b-851d-618c9654116c-istio-envoy\") pod \"43983e9f-061a-437b-851d-618c9654116c\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") "
Sep 17 19:23:18.566698 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.566687    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43983e9f-061a-437b-851d-618c9654116c-istio-podinfo\") pod \"43983e9f-061a-437b-851d-618c9654116c\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") "
Sep 17 19:23:18.566798 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.566763    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43983e9f-061a-437b-851d-618c9654116c-istio-data\") pod \"43983e9f-061a-437b-851d-618c9654116c\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") "
Sep 17 19:23:18.566798 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.566789    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hj4qj\" (UniqueName: \"kubernetes.io/projected/43983e9f-061a-437b-851d-618c9654116c-kube-api-access-hj4qj\") pod \"43983e9f-061a-437b-851d-618c9654116c\" (UID: \"43983e9f-061a-437b-851d-618c9654116c\") "
Sep 17 19:23:18.566913 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.566821    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43983e9f-061a-437b-851d-618c9654116c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "43983e9f-061a-437b-851d-618c9654116c" (UID: "43983e9f-061a-437b-851d-618c9654116c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:23:18.567085 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.567034    2588 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43983e9f-061a-437b-851d-618c9654116c-workload-socket\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:23:18.567085 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.567054    2588 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43983e9f-061a-437b-851d-618c9654116c-istiod-ca-cert\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:23:18.567085 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.567070    2588 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43983e9f-061a-437b-851d-618c9654116c-workload-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:23:18.567085 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.567083    2588 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43983e9f-061a-437b-851d-618c9654116c-credential-socket\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:23:18.567360 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.567225    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43983e9f-061a-437b-851d-618c9654116c-istio-data" (OuterVolumeSpecName: "istio-data") pod "43983e9f-061a-437b-851d-618c9654116c" (UID: "43983e9f-061a-437b-851d-618c9654116c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:23:18.568837 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.568809    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43983e9f-061a-437b-851d-618c9654116c-istio-token" (OuterVolumeSpecName: "istio-token") pod "43983e9f-061a-437b-851d-618c9654116c" (UID: "43983e9f-061a-437b-851d-618c9654116c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:23:18.569260 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.569226    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43983e9f-061a-437b-851d-618c9654116c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "43983e9f-061a-437b-851d-618c9654116c" (UID: "43983e9f-061a-437b-851d-618c9654116c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:23:18.569353 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.569317    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43983e9f-061a-437b-851d-618c9654116c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "43983e9f-061a-437b-851d-618c9654116c" (UID: "43983e9f-061a-437b-851d-618c9654116c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 19:23:18.569353 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.569328    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43983e9f-061a-437b-851d-618c9654116c-kube-api-access-hj4qj" (OuterVolumeSpecName: "kube-api-access-hj4qj") pod "43983e9f-061a-437b-851d-618c9654116c" (UID: "43983e9f-061a-437b-851d-618c9654116c"). InnerVolumeSpecName "kube-api-access-hj4qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:23:18.668295 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.668253    2588 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43983e9f-061a-437b-851d-618c9654116c-istio-data\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:23:18.668295 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.668281    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj4qj\" (UniqueName: \"kubernetes.io/projected/43983e9f-061a-437b-851d-618c9654116c-kube-api-access-hj4qj\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:23:18.668295 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.668291    2588 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43983e9f-061a-437b-851d-618c9654116c-istio-token\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:23:18.668295 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.668300    2588 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43983e9f-061a-437b-851d-618c9654116c-istio-envoy\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:23:18.668295 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.668308    2588 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43983e9f-061a-437b-851d-618c9654116c-istio-podinfo\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:23:18.834220 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:18.834185    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"]
Sep 17 19:23:18.838714 ip-10-0-136-132 kubenswrapper[2588]: W0917 19:23:18.838688    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafaaf67b_a3f9_4500_8d61_9ad0a5b2258f.slice/crio-ac494e69dd1507edcbb8b79a77909f5b0c1ed673e2a4af316ec2c0fb09e63412 WatchSource:0}: Error finding container ac494e69dd1507edcbb8b79a77909f5b0c1ed673e2a4af316ec2c0fb09e63412: Status 404 returned error can't find the container with id ac494e69dd1507edcbb8b79a77909f5b0c1ed673e2a4af316ec2c0fb09e63412
Sep 17 19:23:19.392555 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:19.392511    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"
Sep 17 19:23:19.393020 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:19.392586    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr" event={"ID":"43983e9f-061a-437b-851d-618c9654116c","Type":"ContainerDied","Data":"2c9a5dcb1a0429eff2f4ec7d905b1265cfee19938bf3ff6f3dfa9203c224dd11"}
Sep 17 19:23:19.393020 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:19.392985    2588 scope.go:117] "RemoveContainer" containerID="516037291d71a743b329ad54c67c07c508ba9346881eba90bf9f83070b31909a"
Sep 17 19:23:19.394863 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:19.394838    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7" event={"ID":"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f","Type":"ContainerStarted","Data":"b9b5f7030b79542250679f12aba7d1ad01c7e7b65efb1b6f6fe3b8ebce5c2c1f"}
Sep 17 19:23:19.394946 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:19.394890    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7" event={"ID":"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f","Type":"ContainerStarted","Data":"ac494e69dd1507edcbb8b79a77909f5b0c1ed673e2a4af316ec2c0fb09e63412"}
Sep 17 19:23:19.434517 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:19.434489    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"]
Sep 17 19:23:19.438565 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:19.438543    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-844b774c6-zgpkr"]
Sep 17 19:23:19.506057 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:19.506032    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43983e9f-061a-437b-851d-618c9654116c" path="/var/lib/kubelet/pods/43983e9f-061a-437b-851d-618c9654116c/volumes"
Sep 17 19:23:21.407446 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:21.407416    2588 generic.go:356] "Generic (PLEG): container finished" podID="afaaf67b-a3f9-4500-8d61-9ad0a5b2258f" containerID="b9b5f7030b79542250679f12aba7d1ad01c7e7b65efb1b6f6fe3b8ebce5c2c1f" exitCode=0
Sep 17 19:23:21.407866 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:21.407493    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7" event={"ID":"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f","Type":"ContainerDied","Data":"b9b5f7030b79542250679f12aba7d1ad01c7e7b65efb1b6f6fe3b8ebce5c2c1f"}
Sep 17 19:23:22.414201 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:22.414163    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7" event={"ID":"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f","Type":"ContainerStarted","Data":"5cb3f270a1e3d547cc2fe3be82a5dadc43019f554e36d8f3079013ddc1eba0da"}
Sep 17 19:23:22.440513 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:22.440453    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7" podStartSLOduration=2.304019801 podStartE2EDuration="4.440432436s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.136412635s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:23:22.437085036 +0000 UTC m=+3239.288090434" watchObservedRunningTime="2026-09-17 19:23:22.440432436 +0000 UTC m=+3239.291437836"
Sep 17 19:23:22.521788 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:22.521758    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"
Sep 17 19:23:28.521147 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:28.521105    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"
Sep 17 19:23:28.522754 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:28.522728    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7" podUID="afaaf67b-a3f9-4500-8d61-9ad0a5b2258f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 19:23:33.462857 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:33.462825    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:34.916900 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:34.916865    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:36.397254 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:36.397225    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:38.521329 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:38.521290    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7" podUID="afaaf67b-a3f9-4500-8d61-9ad0a5b2258f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 19:23:41.964363 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:41.964324    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:48.521327 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:48.521274    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7" podUID="afaaf67b-a3f9-4500-8d61-9ad0a5b2258f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 19:23:50.810903 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:50.810870    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:53.550316 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:53.550276    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:53.662781 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:53.662750    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:53.764819 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:53.764785    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:53.864502 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:53.864407    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:53.965827 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:53.965784    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:56.209485 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:56.209432    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:56.309006 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:56.308968    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:56.457505 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:56.457461    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:56.525362 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:56.525249    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:57.015502 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:57.015429    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:58.521875 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:58.521813    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7" podUID="afaaf67b-a3f9-4500-8d61-9ad0a5b2258f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 19:23:59.505449 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:59.505383    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:59.604867 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:59.604828    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:59.958357 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:23:59.958326    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:24:05.527835 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:24:05.527803    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:24:05.605600 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:24:05.605561    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:24:06.694307 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:24:06.694270    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:24:06.771856 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:24:06.771819    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:24:08.521640 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:24:08.521597    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7" podUID="afaaf67b-a3f9-4500-8d61-9ad0a5b2258f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 19:24:18.521505 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:24:18.521466    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7" podUID="afaaf67b-a3f9-4500-8d61-9ad0a5b2258f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 19:24:25.714444 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:24:25.714391    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xlklb_07ee31e0-43bb-4940-837d-969bd840492b/console-operator/1.log"
Sep 17 19:24:25.720401 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:24:25.720378    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xlklb_07ee31e0-43bb-4940-837d-969bd840492b/console-operator/1.log"
Sep 17 19:24:25.722429 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:24:25.722406    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovn-acl-logging/0.log"
Sep 17 19:24:25.728406 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:24:25.728384    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovn-acl-logging/0.log"
Sep 17 19:24:28.181693 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:24:28.181653    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:24:28.522346 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:24:28.522250    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7" podUID="afaaf67b-a3f9-4500-8d61-9ad0a5b2258f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 19:24:38.522393 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:24:38.522293    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7" podUID="afaaf67b-a3f9-4500-8d61-9ad0a5b2258f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 19:24:48.521520 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:24:48.521462    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7" podUID="afaaf67b-a3f9-4500-8d61-9ad0a5b2258f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 19:24:58.521900 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:24:58.521847    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7" podUID="afaaf67b-a3f9-4500-8d61-9ad0a5b2258f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 19:25:08.521771 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:08.521726    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7" podUID="afaaf67b-a3f9-4500-8d61-9ad0a5b2258f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 19:25:18.531208 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:18.531171    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"
Sep 17 19:25:18.539097 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:18.539067    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"
Sep 17 19:25:23.433736 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:23.433704    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"]
Sep 17 19:25:23.451847 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:23.451817    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:25:29.113390 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:29.113348    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:25:29.193389 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:29.193350    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:25:38.456037 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:38.455997    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7" podUID="afaaf67b-a3f9-4500-8d61-9ad0a5b2258f" containerName="main" containerID="cri-o://5cb3f270a1e3d547cc2fe3be82a5dadc43019f554e36d8f3079013ddc1eba0da" gracePeriod=45
Sep 17 19:25:39.194483 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:39.194455    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"
Sep 17 19:25:39.286518 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:39.286439    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-kserve-provision-location\") pod \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\" (UID: \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\") "
Sep 17 19:25:39.286518 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:39.286473    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7fr9\" (UniqueName: \"kubernetes.io/projected/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-kube-api-access-m7fr9\") pod \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\" (UID: \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\") "
Sep 17 19:25:39.286518 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:39.286506    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-home\") pod \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\" (UID: \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\") "
Sep 17 19:25:39.286518 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:39.286585    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-model-cache\") pod \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\" (UID: \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\") "
Sep 17 19:25:39.286518 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:39.286635    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-cabundle-cert\") pod \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\" (UID: \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\") "
Sep 17 19:25:39.286518 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:39.286661    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-tls-certs\") pod \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\" (UID: \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\") "
Sep 17 19:25:39.286518 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:39.286684    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-tmp-dir\") pod \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\" (UID: \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\") "
Sep 17 19:25:39.287210 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:39.286718    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-dshm\") pod \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\" (UID: \"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f\") "
Sep 17 19:25:39.287210 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:39.287056    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "afaaf67b-a3f9-4500-8d61-9ad0a5b2258f" (UID: "afaaf67b-a3f9-4500-8d61-9ad0a5b2258f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:25:39.287210 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:39.287140    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-model-cache" (OuterVolumeSpecName: "model-cache") pod "afaaf67b-a3f9-4500-8d61-9ad0a5b2258f" (UID: "afaaf67b-a3f9-4500-8d61-9ad0a5b2258f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:25:39.287364 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:39.287279    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-home" (OuterVolumeSpecName: "home") pod "afaaf67b-a3f9-4500-8d61-9ad0a5b2258f" (UID: "afaaf67b-a3f9-4500-8d61-9ad0a5b2258f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:25:39.289073 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:39.289041    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-dshm" (OuterVolumeSpecName: "dshm") pod "afaaf67b-a3f9-4500-8d61-9ad0a5b2258f" (UID: "afaaf67b-a3f9-4500-8d61-9ad0a5b2258f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:25:39.289202 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:39.289160    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-kube-api-access-m7fr9" (OuterVolumeSpecName: "kube-api-access-m7fr9") pod "afaaf67b-a3f9-4500-8d61-9ad0a5b2258f" (UID: "afaaf67b-a3f9-4500-8d61-9ad0a5b2258f"). InnerVolumeSpecName "kube-api-access-m7fr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:25:39.289286 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:39.289266    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "afaaf67b-a3f9-4500-8d61-9ad0a5b2258f" (UID: "afaaf67b-a3f9-4500-8d61-9ad0a5b2258f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:25:39.300041 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:39.300013    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "afaaf67b-a3f9-4500-8d61-9ad0a5b2258f" (UID: "afaaf67b-a3f9-4500-8d61-9ad0a5b2258f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:25:39.311834 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:39.311809    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "afaaf67b-a3f9-4500-8d61-9ad0a5b2258f" (UID: "afaaf67b-a3f9-4500-8d61-9ad0a5b2258f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:25:39.387838 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:39.387810    2588 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-cabundle-cert\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:25:39.387838 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:39.387836    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:25:39.387989 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:39.387845    2588 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-tmp-dir\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:25:39.387989 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:39.387853    2588 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-dshm\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:25:39.387989 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:39.387861    2588 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-kserve-provision-location\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:25:39.387989 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:39.387871    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7fr9\" (UniqueName: \"kubernetes.io/projected/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-kube-api-access-m7fr9\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:25:39.387989 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:39.387879    2588 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-home\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:25:39.387989 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:39.387888    2588 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f-model-cache\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:25:40.005901 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:40.005867    2588 generic.go:356] "Generic (PLEG): container finished" podID="afaaf67b-a3f9-4500-8d61-9ad0a5b2258f" containerID="5cb3f270a1e3d547cc2fe3be82a5dadc43019f554e36d8f3079013ddc1eba0da" exitCode=0
Sep 17 19:25:40.006361 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:40.005929    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"
Sep 17 19:25:40.006361 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:40.005929    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7" event={"ID":"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f","Type":"ContainerDied","Data":"5cb3f270a1e3d547cc2fe3be82a5dadc43019f554e36d8f3079013ddc1eba0da"}
Sep 17 19:25:40.006361 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:40.006067    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7" event={"ID":"afaaf67b-a3f9-4500-8d61-9ad0a5b2258f","Type":"ContainerDied","Data":"ac494e69dd1507edcbb8b79a77909f5b0c1ed673e2a4af316ec2c0fb09e63412"}
Sep 17 19:25:40.006361 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:40.006088    2588 scope.go:117] "RemoveContainer" containerID="5cb3f270a1e3d547cc2fe3be82a5dadc43019f554e36d8f3079013ddc1eba0da"
Sep 17 19:25:40.014477 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:40.014449    2588 scope.go:117] "RemoveContainer" containerID="b9b5f7030b79542250679f12aba7d1ad01c7e7b65efb1b6f6fe3b8ebce5c2c1f"
Sep 17 19:25:40.021922 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:40.021903    2588 scope.go:117] "RemoveContainer" containerID="5cb3f270a1e3d547cc2fe3be82a5dadc43019f554e36d8f3079013ddc1eba0da"
Sep 17 19:25:40.022181 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:25:40.022161    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cb3f270a1e3d547cc2fe3be82a5dadc43019f554e36d8f3079013ddc1eba0da\": container with ID starting with 5cb3f270a1e3d547cc2fe3be82a5dadc43019f554e36d8f3079013ddc1eba0da not found: ID does not exist" containerID="5cb3f270a1e3d547cc2fe3be82a5dadc43019f554e36d8f3079013ddc1eba0da"
Sep 17 19:25:40.022232 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:40.022189    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cb3f270a1e3d547cc2fe3be82a5dadc43019f554e36d8f3079013ddc1eba0da"} err="failed to get container status \"5cb3f270a1e3d547cc2fe3be82a5dadc43019f554e36d8f3079013ddc1eba0da\": rpc error: code = NotFound desc = could not find container \"5cb3f270a1e3d547cc2fe3be82a5dadc43019f554e36d8f3079013ddc1eba0da\": container with ID starting with 5cb3f270a1e3d547cc2fe3be82a5dadc43019f554e36d8f3079013ddc1eba0da not found: ID does not exist"
Sep 17 19:25:40.022232 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:40.022205    2588 scope.go:117] "RemoveContainer" containerID="b9b5f7030b79542250679f12aba7d1ad01c7e7b65efb1b6f6fe3b8ebce5c2c1f"
Sep 17 19:25:40.022439 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:25:40.022421    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9b5f7030b79542250679f12aba7d1ad01c7e7b65efb1b6f6fe3b8ebce5c2c1f\": container with ID starting with b9b5f7030b79542250679f12aba7d1ad01c7e7b65efb1b6f6fe3b8ebce5c2c1f not found: ID does not exist" containerID="b9b5f7030b79542250679f12aba7d1ad01c7e7b65efb1b6f6fe3b8ebce5c2c1f"
Sep 17 19:25:40.022504 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:40.022444    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9b5f7030b79542250679f12aba7d1ad01c7e7b65efb1b6f6fe3b8ebce5c2c1f"} err="failed to get container status \"b9b5f7030b79542250679f12aba7d1ad01c7e7b65efb1b6f6fe3b8ebce5c2c1f\": rpc error: code = NotFound desc = could not find container \"b9b5f7030b79542250679f12aba7d1ad01c7e7b65efb1b6f6fe3b8ebce5c2c1f\": container with ID starting with b9b5f7030b79542250679f12aba7d1ad01c7e7b65efb1b6f6fe3b8ebce5c2c1f not found: ID does not exist"
Sep 17 19:25:40.025739 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:40.025718    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"]
Sep 17 19:25:40.030431 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:40.030410    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-69847dfd54tvmd7"]
Sep 17 19:25:41.505454 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:41.505421    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afaaf67b-a3f9-4500-8d61-9ad0a5b2258f" path="/var/lib/kubelet/pods/afaaf67b-a3f9-4500-8d61-9ad0a5b2258f/volumes"
Sep 17 19:25:43.451412 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:43.451376    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:25:43.548733 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:43.548700    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:25:49.779969 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:49.779930    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:25:55.440969 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:55.440938    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:25:55.511260 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:55.511228    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:25:56.374830 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.374790    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:25:56.541351 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.541318    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:25:56.677343 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.677238    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:25:56.711256 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.711188    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 19:25:56.711597 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.711582    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43983e9f-061a-437b-851d-618c9654116c" containerName="istio-proxy"
Sep 17 19:25:56.711656 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.711599    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="43983e9f-061a-437b-851d-618c9654116c" containerName="istio-proxy"
Sep 17 19:25:56.711656 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.711609    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43983e9f-061a-437b-851d-618c9654116c" containerName="istio-proxy"
Sep 17 19:25:56.711656 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.711615    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="43983e9f-061a-437b-851d-618c9654116c" containerName="istio-proxy"
Sep 17 19:25:56.711656 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.711624    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afaaf67b-a3f9-4500-8d61-9ad0a5b2258f" containerName="main"
Sep 17 19:25:56.711656 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.711629    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="afaaf67b-a3f9-4500-8d61-9ad0a5b2258f" containerName="main"
Sep 17 19:25:56.711656 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.711643    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afaaf67b-a3f9-4500-8d61-9ad0a5b2258f" containerName="storage-initializer"
Sep 17 19:25:56.711656 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.711650    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="afaaf67b-a3f9-4500-8d61-9ad0a5b2258f" containerName="storage-initializer"
Sep 17 19:25:56.711868 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.711702    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="afaaf67b-a3f9-4500-8d61-9ad0a5b2258f" containerName="main"
Sep 17 19:25:56.711868 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.711711    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="43983e9f-061a-437b-851d-618c9654116c" containerName="istio-proxy"
Sep 17 19:25:56.711868 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.711798    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="43983e9f-061a-437b-851d-618c9654116c" containerName="istio-proxy"
Sep 17 19:25:56.716598 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.716578    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:25:56.719745 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.719727    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 17 19:25:56.720671 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.720647    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-vf24g\""
Sep 17 19:25:56.720780 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.720765    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-kr2sj\""
Sep 17 19:25:56.720851 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.720723    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 17 19:25:56.720851 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.720819    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:25:56.720968 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.720821    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 17 19:25:56.720968 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.720879    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 17 19:25:56.727181 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.727150    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 19:25:56.729596 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.729215    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dbb4124-0218-49e6-9def-192ddc9b29fa-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0dbb4124-0218-49e6-9def-192ddc9b29fa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:25:56.729596 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.729265    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dbb4124-0218-49e6-9def-192ddc9b29fa-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0dbb4124-0218-49e6-9def-192ddc9b29fa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:25:56.729596 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.729314    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0dbb4124-0218-49e6-9def-192ddc9b29fa-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0dbb4124-0218-49e6-9def-192ddc9b29fa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:25:56.729596 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.729346    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4g68\" (UniqueName: \"kubernetes.io/projected/0dbb4124-0218-49e6-9def-192ddc9b29fa-kube-api-access-g4g68\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0dbb4124-0218-49e6-9def-192ddc9b29fa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:25:56.729596 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.729383    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dbb4124-0218-49e6-9def-192ddc9b29fa-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0dbb4124-0218-49e6-9def-192ddc9b29fa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:25:56.729596 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.729406    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dbb4124-0218-49e6-9def-192ddc9b29fa-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0dbb4124-0218-49e6-9def-192ddc9b29fa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:25:56.729596 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.729430    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0dbb4124-0218-49e6-9def-192ddc9b29fa-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0dbb4124-0218-49e6-9def-192ddc9b29fa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:25:56.729596 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.729490    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dbb4124-0218-49e6-9def-192ddc9b29fa-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0dbb4124-0218-49e6-9def-192ddc9b29fa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:25:56.743500 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.743477    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp"]
Sep 17 19:25:56.746733 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.746713    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp"
Sep 17 19:25:56.749217 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.749197    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-q8jgj\""
Sep 17 19:25:56.764007 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.763981    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp"]
Sep 17 19:25:56.809551 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.809502    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:25:56.830376 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.830350    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjlvr\" (UniqueName: \"kubernetes.io/projected/149f96f2-d620-46ae-a7aa-4ea75cf1ee4f-kube-api-access-gjlvr\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp\" (UID: \"149f96f2-d620-46ae-a7aa-4ea75cf1ee4f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp"
Sep 17 19:25:56.830585 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.830397    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dbb4124-0218-49e6-9def-192ddc9b29fa-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0dbb4124-0218-49e6-9def-192ddc9b29fa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:25:56.830585 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.830537    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dbb4124-0218-49e6-9def-192ddc9b29fa-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0dbb4124-0218-49e6-9def-192ddc9b29fa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:25:56.830585 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.830568    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/149f96f2-d620-46ae-a7aa-4ea75cf1ee4f-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp\" (UID: \"149f96f2-d620-46ae-a7aa-4ea75cf1ee4f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp"
Sep 17 19:25:56.830783 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.830598    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dbb4124-0218-49e6-9def-192ddc9b29fa-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0dbb4124-0218-49e6-9def-192ddc9b29fa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:25:56.830783 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.830630    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0dbb4124-0218-49e6-9def-192ddc9b29fa-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0dbb4124-0218-49e6-9def-192ddc9b29fa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:25:56.830783 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.830690    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4g68\" (UniqueName: \"kubernetes.io/projected/0dbb4124-0218-49e6-9def-192ddc9b29fa-kube-api-access-g4g68\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0dbb4124-0218-49e6-9def-192ddc9b29fa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:25:56.830783 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.830730    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dbb4124-0218-49e6-9def-192ddc9b29fa-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0dbb4124-0218-49e6-9def-192ddc9b29fa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:25:56.830783 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.830754    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dbb4124-0218-49e6-9def-192ddc9b29fa-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0dbb4124-0218-49e6-9def-192ddc9b29fa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:25:56.830783 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.830773    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dbb4124-0218-49e6-9def-192ddc9b29fa-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0dbb4124-0218-49e6-9def-192ddc9b29fa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:25:56.831018 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.830778    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0dbb4124-0218-49e6-9def-192ddc9b29fa-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0dbb4124-0218-49e6-9def-192ddc9b29fa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:25:56.831018 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.830812    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dbb4124-0218-49e6-9def-192ddc9b29fa-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0dbb4124-0218-49e6-9def-192ddc9b29fa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:25:56.831018 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.831000    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dbb4124-0218-49e6-9def-192ddc9b29fa-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0dbb4124-0218-49e6-9def-192ddc9b29fa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:25:56.831228 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.831206    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0dbb4124-0218-49e6-9def-192ddc9b29fa-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0dbb4124-0218-49e6-9def-192ddc9b29fa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:25:56.831372 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.831347    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0dbb4124-0218-49e6-9def-192ddc9b29fa-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0dbb4124-0218-49e6-9def-192ddc9b29fa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:25:56.833166 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.833147    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dbb4124-0218-49e6-9def-192ddc9b29fa-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0dbb4124-0218-49e6-9def-192ddc9b29fa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:25:56.833364 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.833345    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dbb4124-0218-49e6-9def-192ddc9b29fa-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0dbb4124-0218-49e6-9def-192ddc9b29fa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:25:56.844181 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.844162    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4g68\" (UniqueName: \"kubernetes.io/projected/0dbb4124-0218-49e6-9def-192ddc9b29fa-kube-api-access-g4g68\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"0dbb4124-0218-49e6-9def-192ddc9b29fa\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:25:56.931510 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.931428    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/149f96f2-d620-46ae-a7aa-4ea75cf1ee4f-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp\" (UID: \"149f96f2-d620-46ae-a7aa-4ea75cf1ee4f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp"
Sep 17 19:25:56.931510 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.931497    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjlvr\" (UniqueName: \"kubernetes.io/projected/149f96f2-d620-46ae-a7aa-4ea75cf1ee4f-kube-api-access-gjlvr\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp\" (UID: \"149f96f2-d620-46ae-a7aa-4ea75cf1ee4f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp"
Sep 17 19:25:56.934133 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.934100    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/149f96f2-d620-46ae-a7aa-4ea75cf1ee4f-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp\" (UID: \"149f96f2-d620-46ae-a7aa-4ea75cf1ee4f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp"
Sep 17 19:25:56.940056 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:56.940030    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjlvr\" (UniqueName: \"kubernetes.io/projected/149f96f2-d620-46ae-a7aa-4ea75cf1ee4f-kube-api-access-gjlvr\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp\" (UID: \"149f96f2-d620-46ae-a7aa-4ea75cf1ee4f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp"
Sep 17 19:25:57.027226 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:57.027191    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:25:57.058664 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:57.058634    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp"
Sep 17 19:25:57.340595 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:57.340562    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 19:25:57.341053 ip-10-0-136-132 kubenswrapper[2588]: W0917 19:25:57.341022    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dbb4124_0218_49e6_9def_192ddc9b29fa.slice/crio-38d16e6bbfe647d843f1f4fff106a72bbd9510dc5fcb6fabac9e3e667622cba5 WatchSource:0}: Error finding container 38d16e6bbfe647d843f1f4fff106a72bbd9510dc5fcb6fabac9e3e667622cba5: Status 404 returned error can't find the container with id 38d16e6bbfe647d843f1f4fff106a72bbd9510dc5fcb6fabac9e3e667622cba5
Sep 17 19:25:57.365966 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:57.365743    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp"]
Sep 17 19:25:57.370433 ip-10-0-136-132 kubenswrapper[2588]: W0917 19:25:57.370401    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod149f96f2_d620_46ae_a7aa_4ea75cf1ee4f.slice/crio-55619d79036a29c931ae5df8cd6e2f612fcc365778410e63b8d60d0314e61ee2 WatchSource:0}: Error finding container 55619d79036a29c931ae5df8cd6e2f612fcc365778410e63b8d60d0314e61ee2: Status 404 returned error can't find the container with id 55619d79036a29c931ae5df8cd6e2f612fcc365778410e63b8d60d0314e61ee2
Sep 17 19:25:58.079064 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:58.079024    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"0dbb4124-0218-49e6-9def-192ddc9b29fa","Type":"ContainerStarted","Data":"9e5be1f39238742c0072da445dbbddeca285d3825189d9d554da644088435f76"}
Sep 17 19:25:58.079554 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:58.079090    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"0dbb4124-0218-49e6-9def-192ddc9b29fa","Type":"ContainerStarted","Data":"38d16e6bbfe647d843f1f4fff106a72bbd9510dc5fcb6fabac9e3e667622cba5"}
Sep 17 19:25:58.080597 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:58.080561    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp" event={"ID":"149f96f2-d620-46ae-a7aa-4ea75cf1ee4f","Type":"ContainerStarted","Data":"83b8bc992059c5731998f10ad87d6cbdb99517a139a4a8fcd0e501ca45f53212"}
Sep 17 19:25:58.080767 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:58.080622    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp" event={"ID":"149f96f2-d620-46ae-a7aa-4ea75cf1ee4f","Type":"ContainerStarted","Data":"55619d79036a29c931ae5df8cd6e2f612fcc365778410e63b8d60d0314e61ee2"}
Sep 17 19:25:58.080767 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:58.080655    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp"
Sep 17 19:25:58.130738 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:25:58.130669    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp" podStartSLOduration=2.1306523 podStartE2EDuration="2.1306523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:25:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:25:58.127547094 +0000 UTC m=+3394.978552490" watchObservedRunningTime="2026-09-17 19:25:58.1306523 +0000 UTC m=+3394.981657736"
Sep 17 19:26:00.092046 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:26:00.092010    2588 generic.go:356] "Generic (PLEG): container finished" podID="0dbb4124-0218-49e6-9def-192ddc9b29fa" containerID="9e5be1f39238742c0072da445dbbddeca285d3825189d9d554da644088435f76" exitCode=0
Sep 17 19:26:00.092046 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:26:00.092049    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"0dbb4124-0218-49e6-9def-192ddc9b29fa","Type":"ContainerDied","Data":"9e5be1f39238742c0072da445dbbddeca285d3825189d9d554da644088435f76"}
Sep 17 19:26:01.098918 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:26:01.098881    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"0dbb4124-0218-49e6-9def-192ddc9b29fa","Type":"ContainerStarted","Data":"e5a89ec9dfad4e620ccd2608d1fa41b115af2d5e400a24cad34486e247b4e7d3"}
Sep 17 19:26:01.121974 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:26:01.121922    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.871772026 podStartE2EDuration="5.121907895s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.250135869s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:25:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:26:01.120599769 +0000 UTC m=+3397.971605171" watchObservedRunningTime="2026-09-17 19:26:01.121907895 +0000 UTC m=+3397.972913294"
Sep 17 19:26:02.027696 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:26:02.027657    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:26:05.030345 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:26:05.030312    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:26:05.094314 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:26:05.094275    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:26:07.028379 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:26:07.028285    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:26:07.030003 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:26:07.029973    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="0dbb4124-0218-49e6-9def-192ddc9b29fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 17 19:26:08.180523 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:26:08.180489    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:26:08.243048 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:26:08.243011    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:26:17.027854 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:26:17.027812    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="0dbb4124-0218-49e6-9def-192ddc9b29fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 17 19:26:27.028019 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:26:27.027966    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="0dbb4124-0218-49e6-9def-192ddc9b29fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 17 19:26:29.088711 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:26:29.088679    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp"
Sep 17 19:26:35.750107 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:26:35.750069    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:26:37.028198 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:26:37.028149    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="0dbb4124-0218-49e6-9def-192ddc9b29fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 17 19:26:47.028268 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:26:47.028214    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="0dbb4124-0218-49e6-9def-192ddc9b29fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 17 19:26:55.849334 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:26:55.849288    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:26:55.943030 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:26:55.942984    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:26:57.028250 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:26:57.028209    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="0dbb4124-0218-49e6-9def-192ddc9b29fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 17 19:27:01.494920 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:01.494889    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:01.565029 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:01.564992    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:01.651796 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:01.651764    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:01.752563 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:01.752443    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:02.062858 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:02.062829    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:04.251259 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:04.251212    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:04.466331 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:04.466296    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6"]
Sep 17 19:27:04.469613 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:04.469592    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6"
Sep 17 19:27:04.472293 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:04.472268    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 17 19:27:04.472448 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:04.472316    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 17 19:27:04.473210 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:04.473177    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-dxtdh\""
Sep 17 19:27:04.473210 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:04.473184    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-s4c6s\""
Sep 17 19:27:04.473396 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:04.473227    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 17 19:27:04.480865 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:04.480841    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6"]
Sep 17 19:27:04.506806 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:04.506720    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:04.516852 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:04.516825    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a77096c1-0f80-425e-a0b2-f16d6be38193-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6\" (UID: \"a77096c1-0f80-425e-a0b2-f16d6be38193\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6"
Sep 17 19:27:04.516960 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:04.516869    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6r7b\" (UniqueName: \"kubernetes.io/projected/a77096c1-0f80-425e-a0b2-f16d6be38193-kube-api-access-h6r7b\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6\" (UID: \"a77096c1-0f80-425e-a0b2-f16d6be38193\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6"
Sep 17 19:27:04.618175 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:04.618140    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a77096c1-0f80-425e-a0b2-f16d6be38193-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6\" (UID: \"a77096c1-0f80-425e-a0b2-f16d6be38193\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6"
Sep 17 19:27:04.618364 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:04.618184    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6r7b\" (UniqueName: \"kubernetes.io/projected/a77096c1-0f80-425e-a0b2-f16d6be38193-kube-api-access-h6r7b\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6\" (UID: \"a77096c1-0f80-425e-a0b2-f16d6be38193\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6"
Sep 17 19:27:04.620770 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:04.620741    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a77096c1-0f80-425e-a0b2-f16d6be38193-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6\" (UID: \"a77096c1-0f80-425e-a0b2-f16d6be38193\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6"
Sep 17 19:27:04.626586 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:04.626562    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6r7b\" (UniqueName: \"kubernetes.io/projected/a77096c1-0f80-425e-a0b2-f16d6be38193-kube-api-access-h6r7b\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6\" (UID: \"a77096c1-0f80-425e-a0b2-f16d6be38193\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6"
Sep 17 19:27:04.781855 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:04.781753    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6"
Sep 17 19:27:05.078264 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:05.078230    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6"]
Sep 17 19:27:05.081176 ip-10-0-136-132 kubenswrapper[2588]: W0917 19:27:05.081146    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda77096c1_0f80_425e_a0b2_f16d6be38193.slice/crio-fda281b1aeaa6cfdaacd374378d3f3eb9402cc8be290a555ec6f9ace3f34ab28 WatchSource:0}: Error finding container fda281b1aeaa6cfdaacd374378d3f3eb9402cc8be290a555ec6f9ace3f34ab28: Status 404 returned error can't find the container with id fda281b1aeaa6cfdaacd374378d3f3eb9402cc8be290a555ec6f9ace3f34ab28
Sep 17 19:27:05.404678 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:05.404575    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6" event={"ID":"a77096c1-0f80-425e-a0b2-f16d6be38193","Type":"ContainerStarted","Data":"79d79ab6e429dfb1f6967332b34fe3cf9610ddad66a94faab6e3437872d29ef2"}
Sep 17 19:27:05.404678 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:05.404664    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6" event={"ID":"a77096c1-0f80-425e-a0b2-f16d6be38193","Type":"ContainerStarted","Data":"fda281b1aeaa6cfdaacd374378d3f3eb9402cc8be290a555ec6f9ace3f34ab28"}
Sep 17 19:27:05.405129 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:05.404703    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6"
Sep 17 19:27:05.424021 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:05.423959    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6" podStartSLOduration=1.423936971 podStartE2EDuration="1.423936971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:27:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:27:05.423792405 +0000 UTC m=+3462.274797806" watchObservedRunningTime="2026-09-17 19:27:05.423936971 +0000 UTC m=+3462.274942370"
Sep 17 19:27:07.028117 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:07.028068    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="0dbb4124-0218-49e6-9def-192ddc9b29fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 17 19:27:07.583779 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:07.583740    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:07.664001 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:07.663967    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:11.548777 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:11.548731    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:17.028470 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:17.028420    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="0dbb4124-0218-49e6-9def-192ddc9b29fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 17 19:27:27.028505 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:27.028455    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="0dbb4124-0218-49e6-9def-192ddc9b29fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 17 19:27:36.410742 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:36.410658    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6"
Sep 17 19:27:37.037539 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:37.037507    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:27:37.045677 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:37.045650    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:27:43.597481 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:43.597442    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:43.637780 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:43.637744    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 19:27:43.640044 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:43.640014    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp"]
Sep 17 19:27:43.698015 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:43.697978    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:49.088408 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:49.088357    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp" podUID="149f96f2-d620-46ae-a7aa-4ea75cf1ee4f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:27:58.658151 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:58.658108    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp" podUID="149f96f2-d620-46ae-a7aa-4ea75cf1ee4f" containerName="main" containerID="cri-o://83b8bc992059c5731998f10ad87d6cbdb99517a139a4a8fcd0e501ca45f53212" gracePeriod=45
Sep 17 19:27:58.660273 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:58.660242    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="0dbb4124-0218-49e6-9def-192ddc9b29fa" containerName="main" containerID="cri-o://e5a89ec9dfad4e620ccd2608d1fa41b115af2d5e400a24cad34486e247b4e7d3" gracePeriod=45
Sep 17 19:27:59.038221 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.038111    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="0dbb4124-0218-49e6-9def-192ddc9b29fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 17 19:27:59.088047 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.088002    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp" podUID="149f96f2-d620-46ae-a7aa-4ea75cf1ee4f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:27:59.400790 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.400764    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:27:59.488018 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.487967    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dbb4124-0218-49e6-9def-192ddc9b29fa-model-cache\") pod \"0dbb4124-0218-49e6-9def-192ddc9b29fa\" (UID: \"0dbb4124-0218-49e6-9def-192ddc9b29fa\") "
Sep 17 19:27:59.488018 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.488022    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dbb4124-0218-49e6-9def-192ddc9b29fa-home\") pod \"0dbb4124-0218-49e6-9def-192ddc9b29fa\" (UID: \"0dbb4124-0218-49e6-9def-192ddc9b29fa\") "
Sep 17 19:27:59.488281 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.488053    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4g68\" (UniqueName: \"kubernetes.io/projected/0dbb4124-0218-49e6-9def-192ddc9b29fa-kube-api-access-g4g68\") pod \"0dbb4124-0218-49e6-9def-192ddc9b29fa\" (UID: \"0dbb4124-0218-49e6-9def-192ddc9b29fa\") "
Sep 17 19:27:59.488281 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.488080    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dbb4124-0218-49e6-9def-192ddc9b29fa-tls-certs\") pod \"0dbb4124-0218-49e6-9def-192ddc9b29fa\" (UID: \"0dbb4124-0218-49e6-9def-192ddc9b29fa\") "
Sep 17 19:27:59.488281 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.488131    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0dbb4124-0218-49e6-9def-192ddc9b29fa-cabundle-cert\") pod \"0dbb4124-0218-49e6-9def-192ddc9b29fa\" (UID: \"0dbb4124-0218-49e6-9def-192ddc9b29fa\") "
Sep 17 19:27:59.488281 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.488261    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dbb4124-0218-49e6-9def-192ddc9b29fa-dshm\") pod \"0dbb4124-0218-49e6-9def-192ddc9b29fa\" (UID: \"0dbb4124-0218-49e6-9def-192ddc9b29fa\") "
Sep 17 19:27:59.488618 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.488308    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0dbb4124-0218-49e6-9def-192ddc9b29fa-kserve-provision-location\") pod \"0dbb4124-0218-49e6-9def-192ddc9b29fa\" (UID: \"0dbb4124-0218-49e6-9def-192ddc9b29fa\") "
Sep 17 19:27:59.488618 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.488337    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dbb4124-0218-49e6-9def-192ddc9b29fa-model-cache" (OuterVolumeSpecName: "model-cache") pod "0dbb4124-0218-49e6-9def-192ddc9b29fa" (UID: "0dbb4124-0218-49e6-9def-192ddc9b29fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:59.488618 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.488395    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dbb4124-0218-49e6-9def-192ddc9b29fa-tmp-dir\") pod \"0dbb4124-0218-49e6-9def-192ddc9b29fa\" (UID: \"0dbb4124-0218-49e6-9def-192ddc9b29fa\") "
Sep 17 19:27:59.488618 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.488518    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dbb4124-0218-49e6-9def-192ddc9b29fa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0dbb4124-0218-49e6-9def-192ddc9b29fa" (UID: "0dbb4124-0218-49e6-9def-192ddc9b29fa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:27:59.488835 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.488668    2588 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0dbb4124-0218-49e6-9def-192ddc9b29fa-cabundle-cert\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:27:59.488835 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.488687    2588 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dbb4124-0218-49e6-9def-192ddc9b29fa-model-cache\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:27:59.488945 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.488886    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dbb4124-0218-49e6-9def-192ddc9b29fa-home" (OuterVolumeSpecName: "home") pod "0dbb4124-0218-49e6-9def-192ddc9b29fa" (UID: "0dbb4124-0218-49e6-9def-192ddc9b29fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:59.490492 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.490465    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dbb4124-0218-49e6-9def-192ddc9b29fa-dshm" (OuterVolumeSpecName: "dshm") pod "0dbb4124-0218-49e6-9def-192ddc9b29fa" (UID: "0dbb4124-0218-49e6-9def-192ddc9b29fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:59.490853 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.490822    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dbb4124-0218-49e6-9def-192ddc9b29fa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0dbb4124-0218-49e6-9def-192ddc9b29fa" (UID: "0dbb4124-0218-49e6-9def-192ddc9b29fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:27:59.490951 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.490863    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dbb4124-0218-49e6-9def-192ddc9b29fa-kube-api-access-g4g68" (OuterVolumeSpecName: "kube-api-access-g4g68") pod "0dbb4124-0218-49e6-9def-192ddc9b29fa" (UID: "0dbb4124-0218-49e6-9def-192ddc9b29fa"). InnerVolumeSpecName "kube-api-access-g4g68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:27:59.507000 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.506969    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dbb4124-0218-49e6-9def-192ddc9b29fa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0dbb4124-0218-49e6-9def-192ddc9b29fa" (UID: "0dbb4124-0218-49e6-9def-192ddc9b29fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:59.512810 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.512784    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dbb4124-0218-49e6-9def-192ddc9b29fa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0dbb4124-0218-49e6-9def-192ddc9b29fa" (UID: "0dbb4124-0218-49e6-9def-192ddc9b29fa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:59.589994 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.589920    2588 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dbb4124-0218-49e6-9def-192ddc9b29fa-tmp-dir\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:27:59.589994 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.589946    2588 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dbb4124-0218-49e6-9def-192ddc9b29fa-home\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:27:59.589994 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.589955    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4g68\" (UniqueName: \"kubernetes.io/projected/0dbb4124-0218-49e6-9def-192ddc9b29fa-kube-api-access-g4g68\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:27:59.589994 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.589963    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dbb4124-0218-49e6-9def-192ddc9b29fa-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:27:59.589994 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.589971    2588 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dbb4124-0218-49e6-9def-192ddc9b29fa-dshm\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:27:59.589994 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.589979    2588 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0dbb4124-0218-49e6-9def-192ddc9b29fa-kserve-provision-location\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:27:59.653637 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.653607    2588 generic.go:356] "Generic (PLEG): container finished" podID="0dbb4124-0218-49e6-9def-192ddc9b29fa" containerID="e5a89ec9dfad4e620ccd2608d1fa41b115af2d5e400a24cad34486e247b4e7d3" exitCode=0
Sep 17 19:27:59.653786 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.653664    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 19:27:59.653786 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.653688    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"0dbb4124-0218-49e6-9def-192ddc9b29fa","Type":"ContainerDied","Data":"e5a89ec9dfad4e620ccd2608d1fa41b115af2d5e400a24cad34486e247b4e7d3"}
Sep 17 19:27:59.653786 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.653742    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"0dbb4124-0218-49e6-9def-192ddc9b29fa","Type":"ContainerDied","Data":"38d16e6bbfe647d843f1f4fff106a72bbd9510dc5fcb6fabac9e3e667622cba5"}
Sep 17 19:27:59.653786 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.653763    2588 scope.go:117] "RemoveContainer" containerID="e5a89ec9dfad4e620ccd2608d1fa41b115af2d5e400a24cad34486e247b4e7d3"
Sep 17 19:27:59.662956 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.662808    2588 scope.go:117] "RemoveContainer" containerID="9e5be1f39238742c0072da445dbbddeca285d3825189d9d554da644088435f76"
Sep 17 19:27:59.672508 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.672488    2588 scope.go:117] "RemoveContainer" containerID="e5a89ec9dfad4e620ccd2608d1fa41b115af2d5e400a24cad34486e247b4e7d3"
Sep 17 19:27:59.672796 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:27:59.672778    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5a89ec9dfad4e620ccd2608d1fa41b115af2d5e400a24cad34486e247b4e7d3\": container with ID starting with e5a89ec9dfad4e620ccd2608d1fa41b115af2d5e400a24cad34486e247b4e7d3 not found: ID does not exist" containerID="e5a89ec9dfad4e620ccd2608d1fa41b115af2d5e400a24cad34486e247b4e7d3"
Sep 17 19:27:59.672848 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.672805    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5a89ec9dfad4e620ccd2608d1fa41b115af2d5e400a24cad34486e247b4e7d3"} err="failed to get container status \"e5a89ec9dfad4e620ccd2608d1fa41b115af2d5e400a24cad34486e247b4e7d3\": rpc error: code = NotFound desc = could not find container \"e5a89ec9dfad4e620ccd2608d1fa41b115af2d5e400a24cad34486e247b4e7d3\": container with ID starting with e5a89ec9dfad4e620ccd2608d1fa41b115af2d5e400a24cad34486e247b4e7d3 not found: ID does not exist"
Sep 17 19:27:59.672848 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.672821    2588 scope.go:117] "RemoveContainer" containerID="9e5be1f39238742c0072da445dbbddeca285d3825189d9d554da644088435f76"
Sep 17 19:27:59.673083 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:27:59.673066    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e5be1f39238742c0072da445dbbddeca285d3825189d9d554da644088435f76\": container with ID starting with 9e5be1f39238742c0072da445dbbddeca285d3825189d9d554da644088435f76 not found: ID does not exist" containerID="9e5be1f39238742c0072da445dbbddeca285d3825189d9d554da644088435f76"
Sep 17 19:27:59.673128 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.673088    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e5be1f39238742c0072da445dbbddeca285d3825189d9d554da644088435f76"} err="failed to get container status \"9e5be1f39238742c0072da445dbbddeca285d3825189d9d554da644088435f76\": rpc error: code = NotFound desc = could not find container \"9e5be1f39238742c0072da445dbbddeca285d3825189d9d554da644088435f76\": container with ID starting with 9e5be1f39238742c0072da445dbbddeca285d3825189d9d554da644088435f76 not found: ID does not exist"
Sep 17 19:27:59.683824 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.683803    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 19:27:59.687187 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:27:59.687166    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 19:28:01.505599 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:01.505569    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dbb4124-0218-49e6-9def-192ddc9b29fa" path="/var/lib/kubelet/pods/0dbb4124-0218-49e6-9def-192ddc9b29fa/volumes"
Sep 17 19:28:09.088338 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:09.088300    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp" podUID="149f96f2-d620-46ae-a7aa-4ea75cf1ee4f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:28:09.088739 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:09.088403    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp"
Sep 17 19:28:19.088069 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:19.088032    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp" podUID="149f96f2-d620-46ae-a7aa-4ea75cf1ee4f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:28:28.783112 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:28.783072    2588 generic.go:356] "Generic (PLEG): container finished" podID="149f96f2-d620-46ae-a7aa-4ea75cf1ee4f" containerID="83b8bc992059c5731998f10ad87d6cbdb99517a139a4a8fcd0e501ca45f53212" exitCode=0
Sep 17 19:28:28.783436 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:28.783138    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp" event={"ID":"149f96f2-d620-46ae-a7aa-4ea75cf1ee4f","Type":"ContainerDied","Data":"83b8bc992059c5731998f10ad87d6cbdb99517a139a4a8fcd0e501ca45f53212"}
Sep 17 19:28:28.906719 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:28.906692    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp"
Sep 17 19:28:29.024665 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:29.024583    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjlvr\" (UniqueName: \"kubernetes.io/projected/149f96f2-d620-46ae-a7aa-4ea75cf1ee4f-kube-api-access-gjlvr\") pod \"149f96f2-d620-46ae-a7aa-4ea75cf1ee4f\" (UID: \"149f96f2-d620-46ae-a7aa-4ea75cf1ee4f\") "
Sep 17 19:28:29.024665 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:29.024622    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/149f96f2-d620-46ae-a7aa-4ea75cf1ee4f-tls-certs\") pod \"149f96f2-d620-46ae-a7aa-4ea75cf1ee4f\" (UID: \"149f96f2-d620-46ae-a7aa-4ea75cf1ee4f\") "
Sep 17 19:28:29.026749 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:29.026719    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/149f96f2-d620-46ae-a7aa-4ea75cf1ee4f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "149f96f2-d620-46ae-a7aa-4ea75cf1ee4f" (UID: "149f96f2-d620-46ae-a7aa-4ea75cf1ee4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:28:29.026842 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:29.026791    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/149f96f2-d620-46ae-a7aa-4ea75cf1ee4f-kube-api-access-gjlvr" (OuterVolumeSpecName: "kube-api-access-gjlvr") pod "149f96f2-d620-46ae-a7aa-4ea75cf1ee4f" (UID: "149f96f2-d620-46ae-a7aa-4ea75cf1ee4f"). InnerVolumeSpecName "kube-api-access-gjlvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:28:29.125514 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:29.125476    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjlvr\" (UniqueName: \"kubernetes.io/projected/149f96f2-d620-46ae-a7aa-4ea75cf1ee4f-kube-api-access-gjlvr\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:28:29.125514 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:29.125503    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/149f96f2-d620-46ae-a7aa-4ea75cf1ee4f-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:28:29.788316 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:29.788231    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp"
Sep 17 19:28:29.788901 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:29.788233    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp" event={"ID":"149f96f2-d620-46ae-a7aa-4ea75cf1ee4f","Type":"ContainerDied","Data":"55619d79036a29c931ae5df8cd6e2f612fcc365778410e63b8d60d0314e61ee2"}
Sep 17 19:28:29.788901 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:29.788370    2588 scope.go:117] "RemoveContainer" containerID="83b8bc992059c5731998f10ad87d6cbdb99517a139a4a8fcd0e501ca45f53212"
Sep 17 19:28:29.804444 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:29.804416    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp"]
Sep 17 19:28:29.808790 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:29.808766    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche2qjkp"]
Sep 17 19:28:31.505464 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:31.505430    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="149f96f2-d620-46ae-a7aa-4ea75cf1ee4f" path="/var/lib/kubelet/pods/149f96f2-d620-46ae-a7aa-4ea75cf1ee4f/volumes"
Sep 17 19:28:33.806371 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:33.806336    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:28:33.919584 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:33.919554    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:28:40.658386 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:40.658353    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:28:42.466046 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:42.465990    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:28:46.339859 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:46.339815    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:28:46.418903 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:46.418875    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:28:48.894552 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:48.894501    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:28:49.142208 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:49.142177    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbd56td"]
Sep 17 19:28:49.142488 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:49.142475    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dbb4124-0218-49e6-9def-192ddc9b29fa" containerName="storage-initializer"
Sep 17 19:28:49.142488 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:49.142488    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dbb4124-0218-49e6-9def-192ddc9b29fa" containerName="storage-initializer"
Sep 17 19:28:49.142615 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:49.142505    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dbb4124-0218-49e6-9def-192ddc9b29fa" containerName="main"
Sep 17 19:28:49.142615 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:49.142510    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dbb4124-0218-49e6-9def-192ddc9b29fa" containerName="main"
Sep 17 19:28:49.142615 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:49.142518    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="149f96f2-d620-46ae-a7aa-4ea75cf1ee4f" containerName="main"
Sep 17 19:28:49.142615 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:49.142536    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="149f96f2-d620-46ae-a7aa-4ea75cf1ee4f" containerName="main"
Sep 17 19:28:49.142615 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:49.142585    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dbb4124-0218-49e6-9def-192ddc9b29fa" containerName="main"
Sep 17 19:28:49.142615 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:49.142595    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="149f96f2-d620-46ae-a7aa-4ea75cf1ee4f" containerName="main"
Sep 17 19:28:49.146823 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:49.146776    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbd56td"
Sep 17 19:28:49.151731 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:49.151704    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 17 19:28:49.152592 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:49.152566    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-tqz5k\""
Sep 17 19:28:49.152726 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:49.152607    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 17 19:28:49.152726 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:49.152615    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 17 19:28:49.152726 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:49.152634    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-99c7g\""
Sep 17 19:28:49.168719 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:49.168696    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbd56td"]
Sep 17 19:28:49.171909 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:49.171888    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmcnk\" (UniqueName: \"kubernetes.io/projected/8545a938-b1d5-4e0d-86b8-2d531f8758db-kube-api-access-fmcnk\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbd56td\" (UID: \"8545a938-b1d5-4e0d-86b8-2d531f8758db\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbd56td"
Sep 17 19:28:49.172022 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:49.171917    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8545a938-b1d5-4e0d-86b8-2d531f8758db-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbd56td\" (UID: \"8545a938-b1d5-4e0d-86b8-2d531f8758db\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbd56td"
Sep 17 19:28:49.199512 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:49.199489    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:28:49.272923 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:49.272887    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmcnk\" (UniqueName: \"kubernetes.io/projected/8545a938-b1d5-4e0d-86b8-2d531f8758db-kube-api-access-fmcnk\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbd56td\" (UID: \"8545a938-b1d5-4e0d-86b8-2d531f8758db\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbd56td"
Sep 17 19:28:49.273061 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:49.272927    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8545a938-b1d5-4e0d-86b8-2d531f8758db-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbd56td\" (UID: \"8545a938-b1d5-4e0d-86b8-2d531f8758db\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbd56td"
Sep 17 19:28:49.275446 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:49.275427    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8545a938-b1d5-4e0d-86b8-2d531f8758db-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbd56td\" (UID: \"8545a938-b1d5-4e0d-86b8-2d531f8758db\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbd56td"
Sep 17 19:28:49.287123 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:49.287104    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmcnk\" (UniqueName: \"kubernetes.io/projected/8545a938-b1d5-4e0d-86b8-2d531f8758db-kube-api-access-fmcnk\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbd56td\" (UID: \"8545a938-b1d5-4e0d-86b8-2d531f8758db\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbd56td"
Sep 17 19:28:49.457230 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:49.457135    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbd56td"
Sep 17 19:28:49.730270 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:49.730246    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbd56td"]
Sep 17 19:28:49.732383 ip-10-0-136-132 kubenswrapper[2588]: W0917 19:28:49.732353    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8545a938_b1d5_4e0d_86b8_2d531f8758db.slice/crio-e7aea08a2345ad7d54f744d7ae9b9938f5dbd9759ea8fd7a534beb3f12aa1c53 WatchSource:0}: Error finding container e7aea08a2345ad7d54f744d7ae9b9938f5dbd9759ea8fd7a534beb3f12aa1c53: Status 404 returned error can't find the container with id e7aea08a2345ad7d54f744d7ae9b9938f5dbd9759ea8fd7a534beb3f12aa1c53
Sep 17 19:28:49.874923 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:49.874882    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbd56td" event={"ID":"8545a938-b1d5-4e0d-86b8-2d531f8758db","Type":"ContainerStarted","Data":"be4f1e910687db3491ee5034e4e5eeec40faa99e567c2be327c4e2a102711d01"}
Sep 17 19:28:49.875086 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:49.874981    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbd56td" event={"ID":"8545a938-b1d5-4e0d-86b8-2d531f8758db","Type":"ContainerStarted","Data":"e7aea08a2345ad7d54f744d7ae9b9938f5dbd9759ea8fd7a534beb3f12aa1c53"}
Sep 17 19:28:49.875086 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:49.875009    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbd56td"
Sep 17 19:28:49.894033 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:49.893979    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbd56td" podStartSLOduration=0.893961095 podStartE2EDuration="893.961095ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:28:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:28:49.89233483 +0000 UTC m=+3566.743340230" watchObservedRunningTime="2026-09-17 19:28:49.893961095 +0000 UTC m=+3566.744966495"
Sep 17 19:28:55.397472 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:28:55.397442    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:29:09.164096 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:09.164008    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6"]
Sep 17 19:29:09.193435 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:09.193407    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:29:09.275031 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:09.274977    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:29:16.411679 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:16.411595    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6" podUID="a77096c1-0f80-425e-a0b2-f16d6be38193" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:29:20.880907 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:20.880873    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbd56td"
Sep 17 19:29:21.094157 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:21.094120    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:29:23.494877 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:23.494842    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbd56td"]
Sep 17 19:29:24.183640 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:24.183601    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:29:24.184374 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:24.184343    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6" podUID="a77096c1-0f80-425e-a0b2-f16d6be38193" containerName="main" containerID="cri-o://79d79ab6e429dfb1f6967332b34fe3cf9610ddad66a94faab6e3437872d29ef2" gracePeriod=45
Sep 17 19:29:25.749604 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:25.749572    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xlklb_07ee31e0-43bb-4940-837d-969bd840492b/console-operator/1.log"
Sep 17 19:29:25.754824 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:25.754802    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovn-acl-logging/0.log"
Sep 17 19:29:25.757359 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:25.757325    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xlklb_07ee31e0-43bb-4940-837d-969bd840492b/console-operator/1.log"
Sep 17 19:29:25.762123 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:25.762101    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovn-acl-logging/0.log"
Sep 17 19:29:26.410998 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:26.410957    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6" podUID="a77096c1-0f80-425e-a0b2-f16d6be38193" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:29:30.881016 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:30.880980    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbd56td" podUID="8545a938-b1d5-4e0d-86b8-2d531f8758db" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:29:36.411196 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:36.411153    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6" podUID="a77096c1-0f80-425e-a0b2-f16d6be38193" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:29:36.411725 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:36.411272    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6"
Sep 17 19:29:38.516266 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:38.516215    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbd56td" podUID="8545a938-b1d5-4e0d-86b8-2d531f8758db" containerName="main" containerID="cri-o://be4f1e910687db3491ee5034e4e5eeec40faa99e567c2be327c4e2a102711d01" gracePeriod=45
Sep 17 19:29:40.881285 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:40.881249    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbd56td" podUID="8545a938-b1d5-4e0d-86b8-2d531f8758db" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:29:46.410842 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:46.410807    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6" podUID="a77096c1-0f80-425e-a0b2-f16d6be38193" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:29:50.881247 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:50.881213    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbd56td" podUID="8545a938-b1d5-4e0d-86b8-2d531f8758db" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:29:50.881678 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:50.881319    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbd56td"
Sep 17 19:29:54.424299 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:54.424274    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6"
Sep 17 19:29:54.473645 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:54.473573    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a77096c1-0f80-425e-a0b2-f16d6be38193-tls-certs\") pod \"a77096c1-0f80-425e-a0b2-f16d6be38193\" (UID: \"a77096c1-0f80-425e-a0b2-f16d6be38193\") "
Sep 17 19:29:54.473645 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:54.473609    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6r7b\" (UniqueName: \"kubernetes.io/projected/a77096c1-0f80-425e-a0b2-f16d6be38193-kube-api-access-h6r7b\") pod \"a77096c1-0f80-425e-a0b2-f16d6be38193\" (UID: \"a77096c1-0f80-425e-a0b2-f16d6be38193\") "
Sep 17 19:29:54.475882 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:54.475856    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a77096c1-0f80-425e-a0b2-f16d6be38193-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a77096c1-0f80-425e-a0b2-f16d6be38193" (UID: "a77096c1-0f80-425e-a0b2-f16d6be38193"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:29:54.475882 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:54.475870    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a77096c1-0f80-425e-a0b2-f16d6be38193-kube-api-access-h6r7b" (OuterVolumeSpecName: "kube-api-access-h6r7b") pod "a77096c1-0f80-425e-a0b2-f16d6be38193" (UID: "a77096c1-0f80-425e-a0b2-f16d6be38193"). InnerVolumeSpecName "kube-api-access-h6r7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:29:54.575170 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:54.575145    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a77096c1-0f80-425e-a0b2-f16d6be38193-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:29:54.575170 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:54.575168    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6r7b\" (UniqueName: \"kubernetes.io/projected/a77096c1-0f80-425e-a0b2-f16d6be38193-kube-api-access-h6r7b\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:29:55.142344 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:55.142309    2588 generic.go:356] "Generic (PLEG): container finished" podID="a77096c1-0f80-425e-a0b2-f16d6be38193" containerID="79d79ab6e429dfb1f6967332b34fe3cf9610ddad66a94faab6e3437872d29ef2" exitCode=0
Sep 17 19:29:55.142579 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:55.142349    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6"
Sep 17 19:29:55.142579 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:55.142353    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6" event={"ID":"a77096c1-0f80-425e-a0b2-f16d6be38193","Type":"ContainerDied","Data":"79d79ab6e429dfb1f6967332b34fe3cf9610ddad66a94faab6e3437872d29ef2"}
Sep 17 19:29:55.142579 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:55.142407    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6" event={"ID":"a77096c1-0f80-425e-a0b2-f16d6be38193","Type":"ContainerDied","Data":"fda281b1aeaa6cfdaacd374378d3f3eb9402cc8be290a555ec6f9ace3f34ab28"}
Sep 17 19:29:55.142579 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:55.142437    2588 scope.go:117] "RemoveContainer" containerID="79d79ab6e429dfb1f6967332b34fe3cf9610ddad66a94faab6e3437872d29ef2"
Sep 17 19:29:55.150895 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:55.150878    2588 scope.go:117] "RemoveContainer" containerID="79d79ab6e429dfb1f6967332b34fe3cf9610ddad66a94faab6e3437872d29ef2"
Sep 17 19:29:55.151154 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:29:55.151136    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79d79ab6e429dfb1f6967332b34fe3cf9610ddad66a94faab6e3437872d29ef2\": container with ID starting with 79d79ab6e429dfb1f6967332b34fe3cf9610ddad66a94faab6e3437872d29ef2 not found: ID does not exist" containerID="79d79ab6e429dfb1f6967332b34fe3cf9610ddad66a94faab6e3437872d29ef2"
Sep 17 19:29:55.151207 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:55.151158    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79d79ab6e429dfb1f6967332b34fe3cf9610ddad66a94faab6e3437872d29ef2"} err="failed to get container status \"79d79ab6e429dfb1f6967332b34fe3cf9610ddad66a94faab6e3437872d29ef2\": rpc error: code = NotFound desc = could not find container \"79d79ab6e429dfb1f6967332b34fe3cf9610ddad66a94faab6e3437872d29ef2\": container with ID starting with 79d79ab6e429dfb1f6967332b34fe3cf9610ddad66a94faab6e3437872d29ef2 not found: ID does not exist"
Sep 17 19:29:55.162497 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:55.162472    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6"]
Sep 17 19:29:55.166269 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:55.166248    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-cflx6"]
Sep 17 19:29:55.505813 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:55.505697    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a77096c1-0f80-425e-a0b2-f16d6be38193" path="/var/lib/kubelet/pods/a77096c1-0f80-425e-a0b2-f16d6be38193/volumes"
Sep 17 19:29:59.256087 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:59.256038    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:29:59.366939 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:29:59.366911    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:30:00.880932 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:00.880895    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbd56td" podUID="8545a938-b1d5-4e0d-86b8-2d531f8758db" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:30:02.327917 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:02.327886    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:30:02.508099 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:02.508066    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn"]
Sep 17 19:30:02.508364 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:02.508350    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a77096c1-0f80-425e-a0b2-f16d6be38193" containerName="main"
Sep 17 19:30:02.508409 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:02.508368    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="a77096c1-0f80-425e-a0b2-f16d6be38193" containerName="main"
Sep 17 19:30:02.508450 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:02.508431    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="a77096c1-0f80-425e-a0b2-f16d6be38193" containerName="main"
Sep 17 19:30:02.511226 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:02.511209    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn"
Sep 17 19:30:02.514649 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:02.514627    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 17 19:30:02.515363 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:02.515339    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 17 19:30:02.515479 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:02.515438    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 17 19:30:02.515597 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:02.515498    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-kmsng\""
Sep 17 19:30:02.515806 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:02.515789    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-8h8z6\""
Sep 17 19:30:02.524134 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:02.524112    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn"]
Sep 17 19:30:02.537904 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:02.537871    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5345feb8-e2d4-49c4-b540-55e71a11fc32-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn\" (UID: \"5345feb8-e2d4-49c4-b540-55e71a11fc32\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn"
Sep 17 19:30:02.538010 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:02.537926    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqqqz\" (UniqueName: \"kubernetes.io/projected/5345feb8-e2d4-49c4-b540-55e71a11fc32-kube-api-access-hqqqz\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn\" (UID: \"5345feb8-e2d4-49c4-b540-55e71a11fc32\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn"
Sep 17 19:30:02.538716 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:02.538691    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:30:02.644457 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:02.640279    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5345feb8-e2d4-49c4-b540-55e71a11fc32-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn\" (UID: \"5345feb8-e2d4-49c4-b540-55e71a11fc32\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn"
Sep 17 19:30:02.644457 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:02.640353    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqqqz\" (UniqueName: \"kubernetes.io/projected/5345feb8-e2d4-49c4-b540-55e71a11fc32-kube-api-access-hqqqz\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn\" (UID: \"5345feb8-e2d4-49c4-b540-55e71a11fc32\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn"
Sep 17 19:30:02.646635 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:02.646608    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5345feb8-e2d4-49c4-b540-55e71a11fc32-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn\" (UID: \"5345feb8-e2d4-49c4-b540-55e71a11fc32\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn"
Sep 17 19:30:02.651059 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:02.651032    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqqqz\" (UniqueName: \"kubernetes.io/projected/5345feb8-e2d4-49c4-b540-55e71a11fc32-kube-api-access-hqqqz\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn\" (UID: \"5345feb8-e2d4-49c4-b540-55e71a11fc32\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn"
Sep 17 19:30:02.821983 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:02.821940    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn"
Sep 17 19:30:03.100885 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:03.100001    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn"]
Sep 17 19:30:03.103608 ip-10-0-136-132 kubenswrapper[2588]: W0917 19:30:03.103521    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5345feb8_e2d4_49c4_b540_55e71a11fc32.slice/crio-e619f4fbcc2da58dae478d34b88ce77a739fba0e48bedc935611cb4e3455a7a2 WatchSource:0}: Error finding container e619f4fbcc2da58dae478d34b88ce77a739fba0e48bedc935611cb4e3455a7a2: Status 404 returned error can't find the container with id e619f4fbcc2da58dae478d34b88ce77a739fba0e48bedc935611cb4e3455a7a2
Sep 17 19:30:03.175269 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:03.175236    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn" event={"ID":"5345feb8-e2d4-49c4-b540-55e71a11fc32","Type":"ContainerStarted","Data":"2d19a3693cffc8b496905b15ae367b438026670bd791cf8a8127a4dd21010764"}
Sep 17 19:30:03.175269 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:03.175285    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn" event={"ID":"5345feb8-e2d4-49c4-b540-55e71a11fc32","Type":"ContainerStarted","Data":"e619f4fbcc2da58dae478d34b88ce77a739fba0e48bedc935611cb4e3455a7a2"}
Sep 17 19:30:03.175556 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:03.175357    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn"
Sep 17 19:30:03.199809 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:03.199749    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn" podStartSLOduration=1.199729438 podStartE2EDuration="1.199729438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:30:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:30:03.19837109 +0000 UTC m=+3640.049376489" watchObservedRunningTime="2026-09-17 19:30:03.199729438 +0000 UTC m=+3640.050734837"
Sep 17 19:30:05.650647 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:05.650613    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:30:08.760966 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:08.760941    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbd56td"
Sep 17 19:30:08.889679 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:08.889646    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmcnk\" (UniqueName: \"kubernetes.io/projected/8545a938-b1d5-4e0d-86b8-2d531f8758db-kube-api-access-fmcnk\") pod \"8545a938-b1d5-4e0d-86b8-2d531f8758db\" (UID: \"8545a938-b1d5-4e0d-86b8-2d531f8758db\") "
Sep 17 19:30:08.889847 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:08.889739    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8545a938-b1d5-4e0d-86b8-2d531f8758db-tls-certs\") pod \"8545a938-b1d5-4e0d-86b8-2d531f8758db\" (UID: \"8545a938-b1d5-4e0d-86b8-2d531f8758db\") "
Sep 17 19:30:08.891936 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:08.891904    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8545a938-b1d5-4e0d-86b8-2d531f8758db-kube-api-access-fmcnk" (OuterVolumeSpecName: "kube-api-access-fmcnk") pod "8545a938-b1d5-4e0d-86b8-2d531f8758db" (UID: "8545a938-b1d5-4e0d-86b8-2d531f8758db"). InnerVolumeSpecName "kube-api-access-fmcnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:30:08.892067 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:08.891935    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8545a938-b1d5-4e0d-86b8-2d531f8758db-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8545a938-b1d5-4e0d-86b8-2d531f8758db" (UID: "8545a938-b1d5-4e0d-86b8-2d531f8758db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:30:08.990985 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:08.990947    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8545a938-b1d5-4e0d-86b8-2d531f8758db-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:30:08.990985 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:08.990986    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmcnk\" (UniqueName: \"kubernetes.io/projected/8545a938-b1d5-4e0d-86b8-2d531f8758db-kube-api-access-fmcnk\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:30:09.199227 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:09.199142    2588 generic.go:356] "Generic (PLEG): container finished" podID="8545a938-b1d5-4e0d-86b8-2d531f8758db" containerID="be4f1e910687db3491ee5034e4e5eeec40faa99e567c2be327c4e2a102711d01" exitCode=0
Sep 17 19:30:09.199227 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:09.199191    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbd56td"
Sep 17 19:30:09.199398 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:09.199224    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbd56td" event={"ID":"8545a938-b1d5-4e0d-86b8-2d531f8758db","Type":"ContainerDied","Data":"be4f1e910687db3491ee5034e4e5eeec40faa99e567c2be327c4e2a102711d01"}
Sep 17 19:30:09.199398 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:09.199299    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbd56td" event={"ID":"8545a938-b1d5-4e0d-86b8-2d531f8758db","Type":"ContainerDied","Data":"e7aea08a2345ad7d54f744d7ae9b9938f5dbd9759ea8fd7a534beb3f12aa1c53"}
Sep 17 19:30:09.199398 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:09.199322    2588 scope.go:117] "RemoveContainer" containerID="be4f1e910687db3491ee5034e4e5eeec40faa99e567c2be327c4e2a102711d01"
Sep 17 19:30:09.207914 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:09.207896    2588 scope.go:117] "RemoveContainer" containerID="be4f1e910687db3491ee5034e4e5eeec40faa99e567c2be327c4e2a102711d01"
Sep 17 19:30:09.208199 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:30:09.208179    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be4f1e910687db3491ee5034e4e5eeec40faa99e567c2be327c4e2a102711d01\": container with ID starting with be4f1e910687db3491ee5034e4e5eeec40faa99e567c2be327c4e2a102711d01 not found: ID does not exist" containerID="be4f1e910687db3491ee5034e4e5eeec40faa99e567c2be327c4e2a102711d01"
Sep 17 19:30:09.208249 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:09.208207    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be4f1e910687db3491ee5034e4e5eeec40faa99e567c2be327c4e2a102711d01"} err="failed to get container status \"be4f1e910687db3491ee5034e4e5eeec40faa99e567c2be327c4e2a102711d01\": rpc error: code = NotFound desc = could not find container \"be4f1e910687db3491ee5034e4e5eeec40faa99e567c2be327c4e2a102711d01\": container with ID starting with be4f1e910687db3491ee5034e4e5eeec40faa99e567c2be327c4e2a102711d01 not found: ID does not exist"
Sep 17 19:30:09.221038 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:09.221018    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbd56td"]
Sep 17 19:30:09.224991 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:09.224970    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbd56td"]
Sep 17 19:30:09.505502 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:09.505419    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8545a938-b1d5-4e0d-86b8-2d531f8758db" path="/var/lib/kubelet/pods/8545a938-b1d5-4e0d-86b8-2d531f8758db/volumes"
Sep 17 19:30:11.141835 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:11.141788    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:30:11.212474 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:11.212440    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:30:12.331259 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:12.331227    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:30:12.423095 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:12.423061    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:30:12.629313 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:12.629240    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:30:18.152817 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:18.152781    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:30:23.420384 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:23.420350    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5"]
Sep 17 19:30:23.422711 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:23.420675    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8545a938-b1d5-4e0d-86b8-2d531f8758db" containerName="main"
Sep 17 19:30:23.422711 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:23.420687    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="8545a938-b1d5-4e0d-86b8-2d531f8758db" containerName="main"
Sep 17 19:30:23.422711 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:23.420745    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="8545a938-b1d5-4e0d-86b8-2d531f8758db" containerName="main"
Sep 17 19:30:23.423554 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:23.423519    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5"
Sep 17 19:30:23.426838 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:23.426816    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 17 19:30:23.426838 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:23.426831    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 17 19:30:23.427044 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:23.426850    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 17 19:30:23.427044 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:23.426906    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-4s2v6\""
Sep 17 19:30:23.433334 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:23.433311    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5"]
Sep 17 19:30:23.502329 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:23.502288    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkm9h\" (UniqueName: \"kubernetes.io/projected/86093548-6f75-410f-9c02-ece33c5f39d1-kube-api-access-zkm9h\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5\" (UID: \"86093548-6f75-410f-9c02-ece33c5f39d1\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5"
Sep 17 19:30:23.502329 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:23.502333    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86093548-6f75-410f-9c02-ece33c5f39d1-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5\" (UID: \"86093548-6f75-410f-9c02-ece33c5f39d1\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5"
Sep 17 19:30:23.502582 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:23.502366    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86093548-6f75-410f-9c02-ece33c5f39d1-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5\" (UID: \"86093548-6f75-410f-9c02-ece33c5f39d1\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5"
Sep 17 19:30:23.502582 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:23.502437    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86093548-6f75-410f-9c02-ece33c5f39d1-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5\" (UID: \"86093548-6f75-410f-9c02-ece33c5f39d1\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5"
Sep 17 19:30:23.502582 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:23.502484    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86093548-6f75-410f-9c02-ece33c5f39d1-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5\" (UID: \"86093548-6f75-410f-9c02-ece33c5f39d1\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5"
Sep 17 19:30:23.502687 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:23.502585    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86093548-6f75-410f-9c02-ece33c5f39d1-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5\" (UID: \"86093548-6f75-410f-9c02-ece33c5f39d1\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5"
Sep 17 19:30:23.600497 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:23.600468    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:30:23.603659 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:23.603632    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86093548-6f75-410f-9c02-ece33c5f39d1-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5\" (UID: \"86093548-6f75-410f-9c02-ece33c5f39d1\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5"
Sep 17 19:30:23.603793 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:23.603676    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkm9h\" (UniqueName: \"kubernetes.io/projected/86093548-6f75-410f-9c02-ece33c5f39d1-kube-api-access-zkm9h\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5\" (UID: \"86093548-6f75-410f-9c02-ece33c5f39d1\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5"
Sep 17 19:30:23.603793 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:23.603699    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86093548-6f75-410f-9c02-ece33c5f39d1-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5\" (UID: \"86093548-6f75-410f-9c02-ece33c5f39d1\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5"
Sep 17 19:30:23.603793 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:23.603740    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86093548-6f75-410f-9c02-ece33c5f39d1-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5\" (UID: \"86093548-6f75-410f-9c02-ece33c5f39d1\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5"
Sep 17 19:30:23.603793 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:23.603758    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86093548-6f75-410f-9c02-ece33c5f39d1-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5\" (UID: \"86093548-6f75-410f-9c02-ece33c5f39d1\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5"
Sep 17 19:30:23.603793 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:23.603781    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86093548-6f75-410f-9c02-ece33c5f39d1-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5\" (UID: \"86093548-6f75-410f-9c02-ece33c5f39d1\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5"
Sep 17 19:30:23.604067 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:23.604026    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86093548-6f75-410f-9c02-ece33c5f39d1-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5\" (UID: \"86093548-6f75-410f-9c02-ece33c5f39d1\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5"
Sep 17 19:30:23.604132 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:23.604112    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86093548-6f75-410f-9c02-ece33c5f39d1-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5\" (UID: \"86093548-6f75-410f-9c02-ece33c5f39d1\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5"
Sep 17 19:30:23.604182 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:23.604129    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86093548-6f75-410f-9c02-ece33c5f39d1-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5\" (UID: \"86093548-6f75-410f-9c02-ece33c5f39d1\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5"
Sep 17 19:30:23.606076 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:23.606050    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86093548-6f75-410f-9c02-ece33c5f39d1-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5\" (UID: \"86093548-6f75-410f-9c02-ece33c5f39d1\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5"
Sep 17 19:30:23.606327 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:23.606310    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86093548-6f75-410f-9c02-ece33c5f39d1-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5\" (UID: \"86093548-6f75-410f-9c02-ece33c5f39d1\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5"
Sep 17 19:30:23.611620 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:23.611601    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkm9h\" (UniqueName: \"kubernetes.io/projected/86093548-6f75-410f-9c02-ece33c5f39d1-kube-api-access-zkm9h\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5\" (UID: \"86093548-6f75-410f-9c02-ece33c5f39d1\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5"
Sep 17 19:30:23.733460 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:23.733380    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5"
Sep 17 19:30:23.870998 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:23.870968    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:30:24.004706 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:24.004676    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5"]
Sep 17 19:30:24.007170 ip-10-0-136-132 kubenswrapper[2588]: W0917 19:30:24.007138    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86093548_6f75_410f_9c02_ece33c5f39d1.slice/crio-1ce978a3de1b0a603f0da679e73b4ceba21a274c44baaaf9942f0392a4c3aba9 WatchSource:0}: Error finding container 1ce978a3de1b0a603f0da679e73b4ceba21a274c44baaaf9942f0392a4c3aba9: Status 404 returned error can't find the container with id 1ce978a3de1b0a603f0da679e73b4ceba21a274c44baaaf9942f0392a4c3aba9
Sep 17 19:30:24.260213 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:24.260124    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5" event={"ID":"86093548-6f75-410f-9c02-ece33c5f39d1","Type":"ContainerStarted","Data":"593616b23afca523876808461215e872d55c460bf5c24a415484ef53af6d445b"}
Sep 17 19:30:24.260213 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:24.260177    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5" event={"ID":"86093548-6f75-410f-9c02-ece33c5f39d1","Type":"ContainerStarted","Data":"1ce978a3de1b0a603f0da679e73b4ceba21a274c44baaaf9942f0392a4c3aba9"}
Sep 17 19:30:24.278683 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:24.278631    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5" podStartSLOduration=1.278612487 podStartE2EDuration="1.278612487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:30:24.27760289 +0000 UTC m=+3661.128608282" watchObservedRunningTime="2026-09-17 19:30:24.278612487 +0000 UTC m=+3661.129617887"
Sep 17 19:30:24.507262 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:24.507231    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:30:24.609482 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:24.609435    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:30:24.733663 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:24.733614    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5"
Sep 17 19:30:33.734010 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:33.733967    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5"
Sep 17 19:30:33.746966 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:33.746938    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5"
Sep 17 19:30:34.180824 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:34.180794    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn"
Sep 17 19:30:34.311756 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:34.311727    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5"
Sep 17 19:30:47.373658 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:47.373620    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:30:57.399757 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:57.399723    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:30:57.404859 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:57.404816    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5"]
Sep 17 19:30:57.413674 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:30:57.413635    2588 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:30:57.413674 ip-10-0-136-132 kubenswrapper[2588]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:30:57.413674 ip-10-0-136-132 kubenswrapper[2588]:         , stdout: , stderr: , exit code -1
Sep 17 19:30:57.413674 ip-10-0-136-132 kubenswrapper[2588]:  > containerID="593616b23afca523876808461215e872d55c460bf5c24a415484ef53af6d445b" cmd=["/bin/sleep","15"]
Sep 17 19:30:57.413950 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:57.413693    2588 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5" message="" err=<
Sep 17 19:30:57.413950 ip-10-0-136-132 kubenswrapper[2588]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:30:57.413950 ip-10-0-136-132 kubenswrapper[2588]:         , stdout: , stderr: , exit code -1
Sep 17 19:30:57.413950 ip-10-0-136-132 kubenswrapper[2588]:  >
Sep 17 19:30:57.413950 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:30:57.413711    2588 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:30:57.413950 ip-10-0-136-132 kubenswrapper[2588]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:30:57.413950 ip-10-0-136-132 kubenswrapper[2588]:         , stdout: , stderr: , exit code -1
Sep 17 19:30:57.413950 ip-10-0-136-132 kubenswrapper[2588]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5" podUID="86093548-6f75-410f-9c02-ece33c5f39d1" containerName="main" containerID="cri-o://593616b23afca523876808461215e872d55c460bf5c24a415484ef53af6d445b"
Sep 17 19:30:57.413950 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:57.413754    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5" podUID="86093548-6f75-410f-9c02-ece33c5f39d1" containerName="main" containerID="cri-o://593616b23afca523876808461215e872d55c460bf5c24a415484ef53af6d445b" gracePeriod=60
Sep 17 19:30:57.482188 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:57.482150    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:30:58.155333 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:58.155304    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5"
Sep 17 19:30:58.189604 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:58.189572    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86093548-6f75-410f-9c02-ece33c5f39d1-model-cache\") pod \"86093548-6f75-410f-9c02-ece33c5f39d1\" (UID: \"86093548-6f75-410f-9c02-ece33c5f39d1\") "
Sep 17 19:30:58.189789 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:58.189626    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86093548-6f75-410f-9c02-ece33c5f39d1-tmp-dir\") pod \"86093548-6f75-410f-9c02-ece33c5f39d1\" (UID: \"86093548-6f75-410f-9c02-ece33c5f39d1\") "
Sep 17 19:30:58.189789 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:58.189654    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86093548-6f75-410f-9c02-ece33c5f39d1-home\") pod \"86093548-6f75-410f-9c02-ece33c5f39d1\" (UID: \"86093548-6f75-410f-9c02-ece33c5f39d1\") "
Sep 17 19:30:58.189789 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:58.189690    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkm9h\" (UniqueName: \"kubernetes.io/projected/86093548-6f75-410f-9c02-ece33c5f39d1-kube-api-access-zkm9h\") pod \"86093548-6f75-410f-9c02-ece33c5f39d1\" (UID: \"86093548-6f75-410f-9c02-ece33c5f39d1\") "
Sep 17 19:30:58.189789 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:58.189747    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86093548-6f75-410f-9c02-ece33c5f39d1-tls-certs\") pod \"86093548-6f75-410f-9c02-ece33c5f39d1\" (UID: \"86093548-6f75-410f-9c02-ece33c5f39d1\") "
Sep 17 19:30:58.190024 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:58.189791    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86093548-6f75-410f-9c02-ece33c5f39d1-dshm\") pod \"86093548-6f75-410f-9c02-ece33c5f39d1\" (UID: \"86093548-6f75-410f-9c02-ece33c5f39d1\") "
Sep 17 19:30:58.190024 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:58.189927    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86093548-6f75-410f-9c02-ece33c5f39d1-model-cache" (OuterVolumeSpecName: "model-cache") pod "86093548-6f75-410f-9c02-ece33c5f39d1" (UID: "86093548-6f75-410f-9c02-ece33c5f39d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:58.190024 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:58.189992    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86093548-6f75-410f-9c02-ece33c5f39d1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "86093548-6f75-410f-9c02-ece33c5f39d1" (UID: "86093548-6f75-410f-9c02-ece33c5f39d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:58.190024 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:58.190008    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86093548-6f75-410f-9c02-ece33c5f39d1-home" (OuterVolumeSpecName: "home") pod "86093548-6f75-410f-9c02-ece33c5f39d1" (UID: "86093548-6f75-410f-9c02-ece33c5f39d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:58.192337 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:58.192301    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86093548-6f75-410f-9c02-ece33c5f39d1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "86093548-6f75-410f-9c02-ece33c5f39d1" (UID: "86093548-6f75-410f-9c02-ece33c5f39d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:30:58.192460 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:58.192375    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86093548-6f75-410f-9c02-ece33c5f39d1-kube-api-access-zkm9h" (OuterVolumeSpecName: "kube-api-access-zkm9h") pod "86093548-6f75-410f-9c02-ece33c5f39d1" (UID: "86093548-6f75-410f-9c02-ece33c5f39d1"). InnerVolumeSpecName "kube-api-access-zkm9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:30:58.192460 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:58.192392    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86093548-6f75-410f-9c02-ece33c5f39d1-dshm" (OuterVolumeSpecName: "dshm") pod "86093548-6f75-410f-9c02-ece33c5f39d1" (UID: "86093548-6f75-410f-9c02-ece33c5f39d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:58.290798 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:58.290693    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86093548-6f75-410f-9c02-ece33c5f39d1-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:30:58.290798 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:58.290732    2588 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86093548-6f75-410f-9c02-ece33c5f39d1-dshm\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:30:58.290798 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:58.290744    2588 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86093548-6f75-410f-9c02-ece33c5f39d1-model-cache\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:30:58.290798 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:58.290754    2588 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86093548-6f75-410f-9c02-ece33c5f39d1-tmp-dir\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:30:58.290798 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:58.290764    2588 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86093548-6f75-410f-9c02-ece33c5f39d1-home\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:30:58.290798 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:58.290776    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkm9h\" (UniqueName: \"kubernetes.io/projected/86093548-6f75-410f-9c02-ece33c5f39d1-kube-api-access-zkm9h\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:30:58.401356 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:58.401310    2588 generic.go:356] "Generic (PLEG): container finished" podID="86093548-6f75-410f-9c02-ece33c5f39d1" containerID="593616b23afca523876808461215e872d55c460bf5c24a415484ef53af6d445b" exitCode=0
Sep 17 19:30:58.401895 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:58.401363    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5"
Sep 17 19:30:58.401895 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:58.401407    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5" event={"ID":"86093548-6f75-410f-9c02-ece33c5f39d1","Type":"ContainerDied","Data":"593616b23afca523876808461215e872d55c460bf5c24a415484ef53af6d445b"}
Sep 17 19:30:58.401895 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:58.401475    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5" event={"ID":"86093548-6f75-410f-9c02-ece33c5f39d1","Type":"ContainerDied","Data":"1ce978a3de1b0a603f0da679e73b4ceba21a274c44baaaf9942f0392a4c3aba9"}
Sep 17 19:30:58.401895 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:58.401511    2588 scope.go:117] "RemoveContainer" containerID="593616b23afca523876808461215e872d55c460bf5c24a415484ef53af6d445b"
Sep 17 19:30:58.410502 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:58.410474    2588 scope.go:117] "RemoveContainer" containerID="593616b23afca523876808461215e872d55c460bf5c24a415484ef53af6d445b"
Sep 17 19:30:58.410833 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:30:58.410813    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"593616b23afca523876808461215e872d55c460bf5c24a415484ef53af6d445b\": container with ID starting with 593616b23afca523876808461215e872d55c460bf5c24a415484ef53af6d445b not found: ID does not exist" containerID="593616b23afca523876808461215e872d55c460bf5c24a415484ef53af6d445b"
Sep 17 19:30:58.410898 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:58.410842    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"593616b23afca523876808461215e872d55c460bf5c24a415484ef53af6d445b"} err="failed to get container status \"593616b23afca523876808461215e872d55c460bf5c24a415484ef53af6d445b\": rpc error: code = NotFound desc = could not find container \"593616b23afca523876808461215e872d55c460bf5c24a415484ef53af6d445b\": container with ID starting with 593616b23afca523876808461215e872d55c460bf5c24a415484ef53af6d445b not found: ID does not exist"
Sep 17 19:30:58.426799 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:58.426765    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5"]
Sep 17 19:30:58.432381 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:58.432355    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f46sr5"]
Sep 17 19:30:59.510835 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:30:59.510799    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86093548-6f75-410f-9c02-ece33c5f39d1" path="/var/lib/kubelet/pods/86093548-6f75-410f-9c02-ece33c5f39d1/volumes"
Sep 17 19:31:33.490321 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:31:33.490272    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:31:47.479071 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:31:47.479024    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:31:47.572107 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:31:47.572058    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:31:53.637773 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:31:53.637733    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:31:59.395093 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:31:59.395053    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:31:59.509124 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:31:59.509095    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:32:00.490377 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:00.490344    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:32:00.821406 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:00.821363    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f"]
Sep 17 19:32:00.821793 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:00.821773    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86093548-6f75-410f-9c02-ece33c5f39d1" containerName="main"
Sep 17 19:32:00.821793 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:00.821794    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="86093548-6f75-410f-9c02-ece33c5f39d1" containerName="main"
Sep 17 19:32:00.821951 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:00.821870    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="86093548-6f75-410f-9c02-ece33c5f39d1" containerName="main"
Sep 17 19:32:00.824689 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:00.824667    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f"
Sep 17 19:32:00.827050 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:00.827027    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 17 19:32:00.828139 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:00.828105    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-jwjdt\""
Sep 17 19:32:00.828250 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:00.828142    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 17 19:32:00.828250 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:00.828160    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-tstbq\""
Sep 17 19:32:00.828250 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:00.828185    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 17 19:32:00.835925 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:00.835903    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f"]
Sep 17 19:32:00.852637 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:00.852591    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:32:00.959184 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:00.959150    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lr6q\" (UniqueName: \"kubernetes.io/projected/e4debba8-a57a-420e-b944-c58278dce3dd-kube-api-access-8lr6q\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f\" (UID: \"e4debba8-a57a-420e-b944-c58278dce3dd\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f"
Sep 17 19:32:00.959374 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:00.959203    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4debba8-a57a-420e-b944-c58278dce3dd-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f\" (UID: \"e4debba8-a57a-420e-b944-c58278dce3dd\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f"
Sep 17 19:32:01.060108 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:01.060065    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lr6q\" (UniqueName: \"kubernetes.io/projected/e4debba8-a57a-420e-b944-c58278dce3dd-kube-api-access-8lr6q\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f\" (UID: \"e4debba8-a57a-420e-b944-c58278dce3dd\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f"
Sep 17 19:32:01.060326 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:01.060118    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4debba8-a57a-420e-b944-c58278dce3dd-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f\" (UID: \"e4debba8-a57a-420e-b944-c58278dce3dd\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f"
Sep 17 19:32:01.062722 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:01.062694    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4debba8-a57a-420e-b944-c58278dce3dd-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f\" (UID: \"e4debba8-a57a-420e-b944-c58278dce3dd\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f"
Sep 17 19:32:01.071354 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:01.071326    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lr6q\" (UniqueName: \"kubernetes.io/projected/e4debba8-a57a-420e-b944-c58278dce3dd-kube-api-access-8lr6q\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f\" (UID: \"e4debba8-a57a-420e-b944-c58278dce3dd\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f"
Sep 17 19:32:01.135364 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:01.135265    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f"
Sep 17 19:32:01.412669 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:01.412566    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f"]
Sep 17 19:32:01.417107 ip-10-0-136-132 kubenswrapper[2588]: W0917 19:32:01.417065    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4debba8_a57a_420e_b944_c58278dce3dd.slice/crio-8441cff601cee0477b17e58c9d0eb95e9966f0c3a4163244034bb8346263954c WatchSource:0}: Error finding container 8441cff601cee0477b17e58c9d0eb95e9966f0c3a4163244034bb8346263954c: Status 404 returned error can't find the container with id 8441cff601cee0477b17e58c9d0eb95e9966f0c3a4163244034bb8346263954c
Sep 17 19:32:01.643097 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:01.643058    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f" event={"ID":"e4debba8-a57a-420e-b944-c58278dce3dd","Type":"ContainerStarted","Data":"99f96104967052f80a2540ad7218e0fb15e64760b1945b799b444be6840bc479"}
Sep 17 19:32:01.643512 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:01.643121    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f" event={"ID":"e4debba8-a57a-420e-b944-c58278dce3dd","Type":"ContainerStarted","Data":"8441cff601cee0477b17e58c9d0eb95e9966f0c3a4163244034bb8346263954c"}
Sep 17 19:32:01.643512 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:01.643187    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f"
Sep 17 19:32:01.663906 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:01.663776    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f" podStartSLOduration=1.663752384 podStartE2EDuration="1.663752384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:32:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:32:01.660870757 +0000 UTC m=+3758.511876158" watchObservedRunningTime="2026-09-17 19:32:01.663752384 +0000 UTC m=+3758.514757785"
Sep 17 19:32:05.052240 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:05.052202    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"]
Sep 17 19:32:05.055692 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:05.055670    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"
Sep 17 19:32:05.058436 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:05.058414    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 17 19:32:05.058746 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:05.058728    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:32:05.084732 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:05.084695    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"]
Sep 17 19:32:05.193357 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:05.193313    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8e06870-7527-4eba-93dd-eae29afe8cdc-tmp-dir\") pod \"prestop-hook-test-kserve-6894dfb696-xxggb\" (UID: \"b8e06870-7527-4eba-93dd-eae29afe8cdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"
Sep 17 19:32:05.193357 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:05.193364    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8e06870-7527-4eba-93dd-eae29afe8cdc-home\") pod \"prestop-hook-test-kserve-6894dfb696-xxggb\" (UID: \"b8e06870-7527-4eba-93dd-eae29afe8cdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"
Sep 17 19:32:05.193672 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:05.193390    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8e06870-7527-4eba-93dd-eae29afe8cdc-model-cache\") pod \"prestop-hook-test-kserve-6894dfb696-xxggb\" (UID: \"b8e06870-7527-4eba-93dd-eae29afe8cdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"
Sep 17 19:32:05.193672 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:05.193416    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klnq7\" (UniqueName: \"kubernetes.io/projected/b8e06870-7527-4eba-93dd-eae29afe8cdc-kube-api-access-klnq7\") pod \"prestop-hook-test-kserve-6894dfb696-xxggb\" (UID: \"b8e06870-7527-4eba-93dd-eae29afe8cdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"
Sep 17 19:32:05.193672 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:05.193464    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8e06870-7527-4eba-93dd-eae29afe8cdc-kserve-provision-location\") pod \"prestop-hook-test-kserve-6894dfb696-xxggb\" (UID: \"b8e06870-7527-4eba-93dd-eae29afe8cdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"
Sep 17 19:32:05.193672 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:05.193499    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8e06870-7527-4eba-93dd-eae29afe8cdc-tls-certs\") pod \"prestop-hook-test-kserve-6894dfb696-xxggb\" (UID: \"b8e06870-7527-4eba-93dd-eae29afe8cdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"
Sep 17 19:32:05.193672 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:05.193558    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8e06870-7527-4eba-93dd-eae29afe8cdc-cabundle-cert\") pod \"prestop-hook-test-kserve-6894dfb696-xxggb\" (UID: \"b8e06870-7527-4eba-93dd-eae29afe8cdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"
Sep 17 19:32:05.193672 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:05.193592    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8e06870-7527-4eba-93dd-eae29afe8cdc-dshm\") pod \"prestop-hook-test-kserve-6894dfb696-xxggb\" (UID: \"b8e06870-7527-4eba-93dd-eae29afe8cdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"
Sep 17 19:32:05.294884 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:05.294837    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8e06870-7527-4eba-93dd-eae29afe8cdc-dshm\") pod \"prestop-hook-test-kserve-6894dfb696-xxggb\" (UID: \"b8e06870-7527-4eba-93dd-eae29afe8cdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"
Sep 17 19:32:05.295079 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:05.294915    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8e06870-7527-4eba-93dd-eae29afe8cdc-tmp-dir\") pod \"prestop-hook-test-kserve-6894dfb696-xxggb\" (UID: \"b8e06870-7527-4eba-93dd-eae29afe8cdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"
Sep 17 19:32:05.295079 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:05.294944    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8e06870-7527-4eba-93dd-eae29afe8cdc-home\") pod \"prestop-hook-test-kserve-6894dfb696-xxggb\" (UID: \"b8e06870-7527-4eba-93dd-eae29afe8cdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"
Sep 17 19:32:05.295079 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:05.294962    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8e06870-7527-4eba-93dd-eae29afe8cdc-model-cache\") pod \"prestop-hook-test-kserve-6894dfb696-xxggb\" (UID: \"b8e06870-7527-4eba-93dd-eae29afe8cdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"
Sep 17 19:32:05.295079 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:05.294980    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klnq7\" (UniqueName: \"kubernetes.io/projected/b8e06870-7527-4eba-93dd-eae29afe8cdc-kube-api-access-klnq7\") pod \"prestop-hook-test-kserve-6894dfb696-xxggb\" (UID: \"b8e06870-7527-4eba-93dd-eae29afe8cdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"
Sep 17 19:32:05.295079 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:05.295027    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8e06870-7527-4eba-93dd-eae29afe8cdc-kserve-provision-location\") pod \"prestop-hook-test-kserve-6894dfb696-xxggb\" (UID: \"b8e06870-7527-4eba-93dd-eae29afe8cdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"
Sep 17 19:32:05.295079 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:05.295076    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8e06870-7527-4eba-93dd-eae29afe8cdc-tls-certs\") pod \"prestop-hook-test-kserve-6894dfb696-xxggb\" (UID: \"b8e06870-7527-4eba-93dd-eae29afe8cdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"
Sep 17 19:32:05.295413 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:05.295105    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8e06870-7527-4eba-93dd-eae29afe8cdc-cabundle-cert\") pod \"prestop-hook-test-kserve-6894dfb696-xxggb\" (UID: \"b8e06870-7527-4eba-93dd-eae29afe8cdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"
Sep 17 19:32:05.295413 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:05.295358    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8e06870-7527-4eba-93dd-eae29afe8cdc-home\") pod \"prestop-hook-test-kserve-6894dfb696-xxggb\" (UID: \"b8e06870-7527-4eba-93dd-eae29afe8cdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"
Sep 17 19:32:05.295413 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:05.295394    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8e06870-7527-4eba-93dd-eae29afe8cdc-tmp-dir\") pod \"prestop-hook-test-kserve-6894dfb696-xxggb\" (UID: \"b8e06870-7527-4eba-93dd-eae29afe8cdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"
Sep 17 19:32:05.295703 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:05.295679    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8e06870-7527-4eba-93dd-eae29afe8cdc-kserve-provision-location\") pod \"prestop-hook-test-kserve-6894dfb696-xxggb\" (UID: \"b8e06870-7527-4eba-93dd-eae29afe8cdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"
Sep 17 19:32:05.295807 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:05.295728    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8e06870-7527-4eba-93dd-eae29afe8cdc-model-cache\") pod \"prestop-hook-test-kserve-6894dfb696-xxggb\" (UID: \"b8e06870-7527-4eba-93dd-eae29afe8cdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"
Sep 17 19:32:05.295807 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:05.295802    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8e06870-7527-4eba-93dd-eae29afe8cdc-cabundle-cert\") pod \"prestop-hook-test-kserve-6894dfb696-xxggb\" (UID: \"b8e06870-7527-4eba-93dd-eae29afe8cdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"
Sep 17 19:32:05.297155 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:05.297131    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8e06870-7527-4eba-93dd-eae29afe8cdc-dshm\") pod \"prestop-hook-test-kserve-6894dfb696-xxggb\" (UID: \"b8e06870-7527-4eba-93dd-eae29afe8cdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"
Sep 17 19:32:05.297626 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:05.297607    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8e06870-7527-4eba-93dd-eae29afe8cdc-tls-certs\") pod \"prestop-hook-test-kserve-6894dfb696-xxggb\" (UID: \"b8e06870-7527-4eba-93dd-eae29afe8cdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"
Sep 17 19:32:05.304910 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:05.304859    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klnq7\" (UniqueName: \"kubernetes.io/projected/b8e06870-7527-4eba-93dd-eae29afe8cdc-kube-api-access-klnq7\") pod \"prestop-hook-test-kserve-6894dfb696-xxggb\" (UID: \"b8e06870-7527-4eba-93dd-eae29afe8cdc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"
Sep 17 19:32:05.366169 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:05.366109    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"
Sep 17 19:32:05.655556 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:05.655450    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"]
Sep 17 19:32:05.658489 ip-10-0-136-132 kubenswrapper[2588]: W0917 19:32:05.658458    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8e06870_7527_4eba_93dd_eae29afe8cdc.slice/crio-9a9e21a1129890a1a4d9220e19fb48eed2f9a4cd63571f4609d991fb13879f61 WatchSource:0}: Error finding container 9a9e21a1129890a1a4d9220e19fb48eed2f9a4cd63571f4609d991fb13879f61: Status 404 returned error can't find the container with id 9a9e21a1129890a1a4d9220e19fb48eed2f9a4cd63571f4609d991fb13879f61
Sep 17 19:32:06.664727 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:06.664690    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb" event={"ID":"b8e06870-7527-4eba-93dd-eae29afe8cdc","Type":"ContainerStarted","Data":"5dec433910ecb39bb85726065afe41598209d92f0f647bd12103bc6f48ca1189"}
Sep 17 19:32:06.665138 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:06.664748    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb" event={"ID":"b8e06870-7527-4eba-93dd-eae29afe8cdc","Type":"ContainerStarted","Data":"9a9e21a1129890a1a4d9220e19fb48eed2f9a4cd63571f4609d991fb13879f61"}
Sep 17 19:32:08.674236 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:08.674197    2588 generic.go:356] "Generic (PLEG): container finished" podID="b8e06870-7527-4eba-93dd-eae29afe8cdc" containerID="5dec433910ecb39bb85726065afe41598209d92f0f647bd12103bc6f48ca1189" exitCode=0
Sep 17 19:32:08.674662 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:08.674255    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb" event={"ID":"b8e06870-7527-4eba-93dd-eae29afe8cdc","Type":"ContainerDied","Data":"5dec433910ecb39bb85726065afe41598209d92f0f647bd12103bc6f48ca1189"}
Sep 17 19:32:09.680030 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:09.679990    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb" event={"ID":"b8e06870-7527-4eba-93dd-eae29afe8cdc","Type":"ContainerStarted","Data":"16b7a5f008a4d27f604768e8ef558384e5e831834b2aa6105f3e1d4d32deff47"}
Sep 17 19:32:09.706361 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:09.706298    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb" podStartSLOduration=2.547555677 podStartE2EDuration="4.706278868s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.158723191s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:32:09.70424655 +0000 UTC m=+3766.555251949" watchObservedRunningTime="2026-09-17 19:32:09.706278868 +0000 UTC m=+3766.557284266"
Sep 17 19:32:10.366402 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:10.366354    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"
Sep 17 19:32:15.366449 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:15.366404    2588 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"
Sep 17 19:32:15.367908 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:15.367879    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb" podUID="b8e06870-7527-4eba-93dd-eae29afe8cdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 17 19:32:17.754758 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:17.754718    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:32:25.366851 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:25.366809    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb" podUID="b8e06870-7527-4eba-93dd-eae29afe8cdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 17 19:32:32.648514 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:32.648480    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f"
Sep 17 19:32:35.367001 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:35.366954    2588 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb" podUID="b8e06870-7527-4eba-93dd-eae29afe8cdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 17 19:32:37.802375 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:37.802337    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:32:37.819342 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:37.819308    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn"]
Sep 17 19:32:37.826787 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:37.826750    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"]
Sep 17 19:32:37.877999 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:37.877967    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:32:39.281210 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:39.281178    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:32:39.285580 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:39.285549    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f"]
Sep 17 19:32:39.376592 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:39.376521    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:32:42.648896 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:42.648854    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f" podUID="e4debba8-a57a-420e-b944-c58278dce3dd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:32:44.181279 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:44.181240    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn" podUID="5345feb8-e2d4-49c4-b540-55e71a11fc32" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:32:48.452880 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:48.452843    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:32:48.516408 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:48.516372    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:32:52.648672 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:52.648627    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f" podUID="e4debba8-a57a-420e-b944-c58278dce3dd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:32:52.839401 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:52.839329    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn" podUID="5345feb8-e2d4-49c4-b540-55e71a11fc32" containerName="main" containerID="cri-o://2d19a3693cffc8b496905b15ae367b438026670bd791cf8a8127a4dd21010764" gracePeriod=45
Sep 17 19:32:52.848700 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:52.848660    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb" podUID="b8e06870-7527-4eba-93dd-eae29afe8cdc" containerName="main" containerID="cri-o://16b7a5f008a4d27f604768e8ef558384e5e831834b2aa6105f3e1d4d32deff47" gracePeriod=45
Sep 17 19:32:54.181092 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.181048    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn" podUID="5345feb8-e2d4-49c4-b540-55e71a11fc32" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:32:54.315549 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.315487    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f" podUID="e4debba8-a57a-420e-b944-c58278dce3dd" containerName="main" containerID="cri-o://99f96104967052f80a2540ad7218e0fb15e64760b1945b799b444be6840bc479" gracePeriod=45
Sep 17 19:32:54.698970 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.698946    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-6894dfb696-xxggb_b8e06870-7527-4eba-93dd-eae29afe8cdc/main/0.log"
Sep 17 19:32:54.701486 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.701467    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"
Sep 17 19:32:54.817670 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.817636    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8e06870-7527-4eba-93dd-eae29afe8cdc-dshm\") pod \"b8e06870-7527-4eba-93dd-eae29afe8cdc\" (UID: \"b8e06870-7527-4eba-93dd-eae29afe8cdc\") "
Sep 17 19:32:54.817835 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.817679    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8e06870-7527-4eba-93dd-eae29afe8cdc-tmp-dir\") pod \"b8e06870-7527-4eba-93dd-eae29afe8cdc\" (UID: \"b8e06870-7527-4eba-93dd-eae29afe8cdc\") "
Sep 17 19:32:54.817835 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.817697    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8e06870-7527-4eba-93dd-eae29afe8cdc-home\") pod \"b8e06870-7527-4eba-93dd-eae29afe8cdc\" (UID: \"b8e06870-7527-4eba-93dd-eae29afe8cdc\") "
Sep 17 19:32:54.817835 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.817726    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8e06870-7527-4eba-93dd-eae29afe8cdc-kserve-provision-location\") pod \"b8e06870-7527-4eba-93dd-eae29afe8cdc\" (UID: \"b8e06870-7527-4eba-93dd-eae29afe8cdc\") "
Sep 17 19:32:54.817835 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.817747    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8e06870-7527-4eba-93dd-eae29afe8cdc-cabundle-cert\") pod \"b8e06870-7527-4eba-93dd-eae29afe8cdc\" (UID: \"b8e06870-7527-4eba-93dd-eae29afe8cdc\") "
Sep 17 19:32:54.817835 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.817767    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8e06870-7527-4eba-93dd-eae29afe8cdc-model-cache\") pod \"b8e06870-7527-4eba-93dd-eae29afe8cdc\" (UID: \"b8e06870-7527-4eba-93dd-eae29afe8cdc\") "
Sep 17 19:32:54.817835 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.817790    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8e06870-7527-4eba-93dd-eae29afe8cdc-tls-certs\") pod \"b8e06870-7527-4eba-93dd-eae29afe8cdc\" (UID: \"b8e06870-7527-4eba-93dd-eae29afe8cdc\") "
Sep 17 19:32:54.818190 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.817847    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klnq7\" (UniqueName: \"kubernetes.io/projected/b8e06870-7527-4eba-93dd-eae29afe8cdc-kube-api-access-klnq7\") pod \"b8e06870-7527-4eba-93dd-eae29afe8cdc\" (UID: \"b8e06870-7527-4eba-93dd-eae29afe8cdc\") "
Sep 17 19:32:54.818190 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.818093    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8e06870-7527-4eba-93dd-eae29afe8cdc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b8e06870-7527-4eba-93dd-eae29afe8cdc" (UID: "b8e06870-7527-4eba-93dd-eae29afe8cdc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:32:54.818190 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.818106    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8e06870-7527-4eba-93dd-eae29afe8cdc-home" (OuterVolumeSpecName: "home") pod "b8e06870-7527-4eba-93dd-eae29afe8cdc" (UID: "b8e06870-7527-4eba-93dd-eae29afe8cdc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:32:54.818190 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.818150    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8e06870-7527-4eba-93dd-eae29afe8cdc-model-cache" (OuterVolumeSpecName: "model-cache") pod "b8e06870-7527-4eba-93dd-eae29afe8cdc" (UID: "b8e06870-7527-4eba-93dd-eae29afe8cdc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:32:54.818399 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.818198    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8e06870-7527-4eba-93dd-eae29afe8cdc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b8e06870-7527-4eba-93dd-eae29afe8cdc" (UID: "b8e06870-7527-4eba-93dd-eae29afe8cdc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:32:54.820234 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.820205    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8e06870-7527-4eba-93dd-eae29afe8cdc-kube-api-access-klnq7" (OuterVolumeSpecName: "kube-api-access-klnq7") pod "b8e06870-7527-4eba-93dd-eae29afe8cdc" (UID: "b8e06870-7527-4eba-93dd-eae29afe8cdc"). InnerVolumeSpecName "kube-api-access-klnq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:32:54.820234 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.820228    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8e06870-7527-4eba-93dd-eae29afe8cdc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b8e06870-7527-4eba-93dd-eae29afe8cdc" (UID: "b8e06870-7527-4eba-93dd-eae29afe8cdc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:32:54.820381 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.820269    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8e06870-7527-4eba-93dd-eae29afe8cdc-dshm" (OuterVolumeSpecName: "dshm") pod "b8e06870-7527-4eba-93dd-eae29afe8cdc" (UID: "b8e06870-7527-4eba-93dd-eae29afe8cdc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:32:54.847303 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.847273    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8e06870-7527-4eba-93dd-eae29afe8cdc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b8e06870-7527-4eba-93dd-eae29afe8cdc" (UID: "b8e06870-7527-4eba-93dd-eae29afe8cdc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:32:54.898391 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.898368    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-6894dfb696-xxggb_b8e06870-7527-4eba-93dd-eae29afe8cdc/main/0.log"
Sep 17 19:32:54.898786 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.898764    2588 generic.go:356] "Generic (PLEG): container finished" podID="b8e06870-7527-4eba-93dd-eae29afe8cdc" containerID="16b7a5f008a4d27f604768e8ef558384e5e831834b2aa6105f3e1d4d32deff47" exitCode=130
Sep 17 19:32:54.898869 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.898813    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"
Sep 17 19:32:54.898869 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.898847    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb" event={"ID":"b8e06870-7527-4eba-93dd-eae29afe8cdc","Type":"ContainerDied","Data":"16b7a5f008a4d27f604768e8ef558384e5e831834b2aa6105f3e1d4d32deff47"}
Sep 17 19:32:54.898954 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.898901    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb" event={"ID":"b8e06870-7527-4eba-93dd-eae29afe8cdc","Type":"ContainerDied","Data":"9a9e21a1129890a1a4d9220e19fb48eed2f9a4cd63571f4609d991fb13879f61"}
Sep 17 19:32:54.898954 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.898924    2588 scope.go:117] "RemoveContainer" containerID="16b7a5f008a4d27f604768e8ef558384e5e831834b2aa6105f3e1d4d32deff47"
Sep 17 19:32:54.907802 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.907784    2588 scope.go:117] "RemoveContainer" containerID="5dec433910ecb39bb85726065afe41598209d92f0f647bd12103bc6f48ca1189"
Sep 17 19:32:54.915037 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.915020    2588 scope.go:117] "RemoveContainer" containerID="16b7a5f008a4d27f604768e8ef558384e5e831834b2aa6105f3e1d4d32deff47"
Sep 17 19:32:54.915324 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:32:54.915305    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16b7a5f008a4d27f604768e8ef558384e5e831834b2aa6105f3e1d4d32deff47\": container with ID starting with 16b7a5f008a4d27f604768e8ef558384e5e831834b2aa6105f3e1d4d32deff47 not found: ID does not exist" containerID="16b7a5f008a4d27f604768e8ef558384e5e831834b2aa6105f3e1d4d32deff47"
Sep 17 19:32:54.915376 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.915335    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16b7a5f008a4d27f604768e8ef558384e5e831834b2aa6105f3e1d4d32deff47"} err="failed to get container status \"16b7a5f008a4d27f604768e8ef558384e5e831834b2aa6105f3e1d4d32deff47\": rpc error: code = NotFound desc = could not find container \"16b7a5f008a4d27f604768e8ef558384e5e831834b2aa6105f3e1d4d32deff47\": container with ID starting with 16b7a5f008a4d27f604768e8ef558384e5e831834b2aa6105f3e1d4d32deff47 not found: ID does not exist"
Sep 17 19:32:54.915376 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.915354    2588 scope.go:117] "RemoveContainer" containerID="5dec433910ecb39bb85726065afe41598209d92f0f647bd12103bc6f48ca1189"
Sep 17 19:32:54.915743 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:32:54.915719    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dec433910ecb39bb85726065afe41598209d92f0f647bd12103bc6f48ca1189\": container with ID starting with 5dec433910ecb39bb85726065afe41598209d92f0f647bd12103bc6f48ca1189 not found: ID does not exist" containerID="5dec433910ecb39bb85726065afe41598209d92f0f647bd12103bc6f48ca1189"
Sep 17 19:32:54.915831 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.915745    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dec433910ecb39bb85726065afe41598209d92f0f647bd12103bc6f48ca1189"} err="failed to get container status \"5dec433910ecb39bb85726065afe41598209d92f0f647bd12103bc6f48ca1189\": rpc error: code = NotFound desc = could not find container \"5dec433910ecb39bb85726065afe41598209d92f0f647bd12103bc6f48ca1189\": container with ID starting with 5dec433910ecb39bb85726065afe41598209d92f0f647bd12103bc6f48ca1189 not found: ID does not exist"
Sep 17 19:32:54.918976 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.918956    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klnq7\" (UniqueName: \"kubernetes.io/projected/b8e06870-7527-4eba-93dd-eae29afe8cdc-kube-api-access-klnq7\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:32:54.918976 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.918976    2588 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8e06870-7527-4eba-93dd-eae29afe8cdc-dshm\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:32:54.919082 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.918986    2588 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8e06870-7527-4eba-93dd-eae29afe8cdc-tmp-dir\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:32:54.919082 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.918995    2588 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8e06870-7527-4eba-93dd-eae29afe8cdc-home\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:32:54.919082 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.919003    2588 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8e06870-7527-4eba-93dd-eae29afe8cdc-kserve-provision-location\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:32:54.919082 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.919011    2588 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8e06870-7527-4eba-93dd-eae29afe8cdc-cabundle-cert\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:32:54.919082 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.919018    2588 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8e06870-7527-4eba-93dd-eae29afe8cdc-model-cache\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:32:54.919082 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.919026    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8e06870-7527-4eba-93dd-eae29afe8cdc-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:32:54.924911 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.924889    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"]
Sep 17 19:32:54.931261 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:54.931240    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xxggb"]
Sep 17 19:32:55.505964 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:32:55.505923    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8e06870-7527-4eba-93dd-eae29afe8cdc" path="/var/lib/kubelet/pods/b8e06870-7527-4eba-93dd-eae29afe8cdc/volumes"
Sep 17 19:33:02.648924 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:02.648881    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f" podUID="e4debba8-a57a-420e-b944-c58278dce3dd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:33:02.649618 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:02.649011    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f"
Sep 17 19:33:04.180871 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:04.180832    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn" podUID="5345feb8-e2d4-49c4-b540-55e71a11fc32" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:33:04.181286 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:04.180935    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn"
Sep 17 19:33:12.649275 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:12.649233    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f" podUID="e4debba8-a57a-420e-b944-c58278dce3dd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:33:14.180896 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:14.180860    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn" podUID="5345feb8-e2d4-49c4-b540-55e71a11fc32" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:33:22.648797 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:22.648748    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f" podUID="e4debba8-a57a-420e-b944-c58278dce3dd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:33:23.018042 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:23.018006    2588 generic.go:356] "Generic (PLEG): container finished" podID="5345feb8-e2d4-49c4-b540-55e71a11fc32" containerID="2d19a3693cffc8b496905b15ae367b438026670bd791cf8a8127a4dd21010764" exitCode=0
Sep 17 19:33:23.018210 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:23.018074    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn" event={"ID":"5345feb8-e2d4-49c4-b540-55e71a11fc32","Type":"ContainerDied","Data":"2d19a3693cffc8b496905b15ae367b438026670bd791cf8a8127a4dd21010764"}
Sep 17 19:33:23.092392 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:23.092367    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn"
Sep 17 19:33:23.151184 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:23.151153    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqqqz\" (UniqueName: \"kubernetes.io/projected/5345feb8-e2d4-49c4-b540-55e71a11fc32-kube-api-access-hqqqz\") pod \"5345feb8-e2d4-49c4-b540-55e71a11fc32\" (UID: \"5345feb8-e2d4-49c4-b540-55e71a11fc32\") "
Sep 17 19:33:23.151353 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:23.151198    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5345feb8-e2d4-49c4-b540-55e71a11fc32-tls-certs\") pod \"5345feb8-e2d4-49c4-b540-55e71a11fc32\" (UID: \"5345feb8-e2d4-49c4-b540-55e71a11fc32\") "
Sep 17 19:33:23.153282 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:23.153252    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5345feb8-e2d4-49c4-b540-55e71a11fc32-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5345feb8-e2d4-49c4-b540-55e71a11fc32" (UID: "5345feb8-e2d4-49c4-b540-55e71a11fc32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:33:23.153494 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:23.153472    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5345feb8-e2d4-49c4-b540-55e71a11fc32-kube-api-access-hqqqz" (OuterVolumeSpecName: "kube-api-access-hqqqz") pod "5345feb8-e2d4-49c4-b540-55e71a11fc32" (UID: "5345feb8-e2d4-49c4-b540-55e71a11fc32"). InnerVolumeSpecName "kube-api-access-hqqqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:33:23.252044 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:23.251965    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqqqz\" (UniqueName: \"kubernetes.io/projected/5345feb8-e2d4-49c4-b540-55e71a11fc32-kube-api-access-hqqqz\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:33:23.252044 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:23.251994    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5345feb8-e2d4-49c4-b540-55e71a11fc32-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:33:24.023667 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:24.023633    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn"
Sep 17 19:33:24.023667 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:24.023658    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn" event={"ID":"5345feb8-e2d4-49c4-b540-55e71a11fc32","Type":"ContainerDied","Data":"e619f4fbcc2da58dae478d34b88ce77a739fba0e48bedc935611cb4e3455a7a2"}
Sep 17 19:33:24.024153 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:24.023721    2588 scope.go:117] "RemoveContainer" containerID="2d19a3693cffc8b496905b15ae367b438026670bd791cf8a8127a4dd21010764"
Sep 17 19:33:24.042043 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:24.042014    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn"]
Sep 17 19:33:24.048005 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:24.047978    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-6ljdn"]
Sep 17 19:33:24.556227 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:24.556200    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f"
Sep 17 19:33:24.662091 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:24.662005    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4debba8-a57a-420e-b944-c58278dce3dd-tls-certs\") pod \"e4debba8-a57a-420e-b944-c58278dce3dd\" (UID: \"e4debba8-a57a-420e-b944-c58278dce3dd\") "
Sep 17 19:33:24.662091 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:24.662054    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lr6q\" (UniqueName: \"kubernetes.io/projected/e4debba8-a57a-420e-b944-c58278dce3dd-kube-api-access-8lr6q\") pod \"e4debba8-a57a-420e-b944-c58278dce3dd\" (UID: \"e4debba8-a57a-420e-b944-c58278dce3dd\") "
Sep 17 19:33:24.664030 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:24.664002    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4debba8-a57a-420e-b944-c58278dce3dd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e4debba8-a57a-420e-b944-c58278dce3dd" (UID: "e4debba8-a57a-420e-b944-c58278dce3dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:33:24.664155 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:24.664053    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4debba8-a57a-420e-b944-c58278dce3dd-kube-api-access-8lr6q" (OuterVolumeSpecName: "kube-api-access-8lr6q") pod "e4debba8-a57a-420e-b944-c58278dce3dd" (UID: "e4debba8-a57a-420e-b944-c58278dce3dd"). InnerVolumeSpecName "kube-api-access-8lr6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:33:24.762930 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:24.762899    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4debba8-a57a-420e-b944-c58278dce3dd-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:33:24.762930 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:24.762923    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lr6q\" (UniqueName: \"kubernetes.io/projected/e4debba8-a57a-420e-b944-c58278dce3dd-kube-api-access-8lr6q\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:33:25.029682 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:25.029590    2588 generic.go:356] "Generic (PLEG): container finished" podID="e4debba8-a57a-420e-b944-c58278dce3dd" containerID="99f96104967052f80a2540ad7218e0fb15e64760b1945b799b444be6840bc479" exitCode=0
Sep 17 19:33:25.029682 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:25.029647    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f"
Sep 17 19:33:25.030211 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:25.029692    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f" event={"ID":"e4debba8-a57a-420e-b944-c58278dce3dd","Type":"ContainerDied","Data":"99f96104967052f80a2540ad7218e0fb15e64760b1945b799b444be6840bc479"}
Sep 17 19:33:25.030211 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:25.029757    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f" event={"ID":"e4debba8-a57a-420e-b944-c58278dce3dd","Type":"ContainerDied","Data":"8441cff601cee0477b17e58c9d0eb95e9966f0c3a4163244034bb8346263954c"}
Sep 17 19:33:25.030211 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:25.029798    2588 scope.go:117] "RemoveContainer" containerID="99f96104967052f80a2540ad7218e0fb15e64760b1945b799b444be6840bc479"
Sep 17 19:33:25.038816 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:25.038796    2588 scope.go:117] "RemoveContainer" containerID="99f96104967052f80a2540ad7218e0fb15e64760b1945b799b444be6840bc479"
Sep 17 19:33:25.039104 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:33:25.039085    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99f96104967052f80a2540ad7218e0fb15e64760b1945b799b444be6840bc479\": container with ID starting with 99f96104967052f80a2540ad7218e0fb15e64760b1945b799b444be6840bc479 not found: ID does not exist" containerID="99f96104967052f80a2540ad7218e0fb15e64760b1945b799b444be6840bc479"
Sep 17 19:33:25.039163 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:25.039113    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99f96104967052f80a2540ad7218e0fb15e64760b1945b799b444be6840bc479"} err="failed to get container status \"99f96104967052f80a2540ad7218e0fb15e64760b1945b799b444be6840bc479\": rpc error: code = NotFound desc = could not find container \"99f96104967052f80a2540ad7218e0fb15e64760b1945b799b444be6840bc479\": container with ID starting with 99f96104967052f80a2540ad7218e0fb15e64760b1945b799b444be6840bc479 not found: ID does not exist"
Sep 17 19:33:25.051788 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:25.051760    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f"]
Sep 17 19:33:25.055662 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:25.055644    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9wvp2f"]
Sep 17 19:33:25.505268 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:25.505235    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5345feb8-e2d4-49c4-b540-55e71a11fc32" path="/var/lib/kubelet/pods/5345feb8-e2d4-49c4-b540-55e71a11fc32/volumes"
Sep 17 19:33:25.505631 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:25.505607    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4debba8-a57a-420e-b944-c58278dce3dd" path="/var/lib/kubelet/pods/e4debba8-a57a-420e-b944-c58278dce3dd/volumes"
Sep 17 19:33:28.592261 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:28.592226    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:33:28.690140 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:28.690107    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:33:28.931015 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:28.930927    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:33:34.856126 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:34.856096    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:33:35.258955 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:35.258870    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:33:40.368176 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:40.368092    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:33:40.439660 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:40.439629    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:33:41.057253 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:41.057214    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:33:41.157069 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:41.157036    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:33:41.351706 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:41.351621    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:33:41.547069 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:41.547032    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9"]
Sep 17 19:33:41.547555 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:41.547337    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8e06870-7527-4eba-93dd-eae29afe8cdc" containerName="storage-initializer"
Sep 17 19:33:41.547555 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:41.547351    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8e06870-7527-4eba-93dd-eae29afe8cdc" containerName="storage-initializer"
Sep 17 19:33:41.547555 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:41.547366    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4debba8-a57a-420e-b944-c58278dce3dd" containerName="main"
Sep 17 19:33:41.547555 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:41.547372    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4debba8-a57a-420e-b944-c58278dce3dd" containerName="main"
Sep 17 19:33:41.547555 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:41.547379    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8e06870-7527-4eba-93dd-eae29afe8cdc" containerName="main"
Sep 17 19:33:41.547555 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:41.547384    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8e06870-7527-4eba-93dd-eae29afe8cdc" containerName="main"
Sep 17 19:33:41.547555 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:41.547399    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5345feb8-e2d4-49c4-b540-55e71a11fc32" containerName="main"
Sep 17 19:33:41.547555 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:41.547404    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="5345feb8-e2d4-49c4-b540-55e71a11fc32" containerName="main"
Sep 17 19:33:41.547555 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:41.547467    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="5345feb8-e2d4-49c4-b540-55e71a11fc32" containerName="main"
Sep 17 19:33:41.547555 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:41.547475    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4debba8-a57a-420e-b944-c58278dce3dd" containerName="main"
Sep 17 19:33:41.547555 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:41.547481    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8e06870-7527-4eba-93dd-eae29afe8cdc" containerName="main"
Sep 17 19:33:41.550447 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:41.550426    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9"
Sep 17 19:33:41.556634 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:41.556610    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 17 19:33:41.557464 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:41.557441    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 17 19:33:41.557606 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:41.557515    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-mwwcn\""
Sep 17 19:33:41.557606 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:41.557558    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 17 19:33:41.557735 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:41.557608    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-9psbc\""
Sep 17 19:33:41.571450 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:41.571429    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9"]
Sep 17 19:33:41.596509 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:41.596484    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56zst\" (UniqueName: \"kubernetes.io/projected/6f8ade4e-1c5f-425e-9173-fe7903f4eeab-kube-api-access-56zst\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9\" (UID: \"6f8ade4e-1c5f-425e-9173-fe7903f4eeab\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9"
Sep 17 19:33:41.596641 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:41.596584    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f8ade4e-1c5f-425e-9173-fe7903f4eeab-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9\" (UID: \"6f8ade4e-1c5f-425e-9173-fe7903f4eeab\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9"
Sep 17 19:33:41.597087 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:41.597070    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:33:41.697545 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:41.697443    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56zst\" (UniqueName: \"kubernetes.io/projected/6f8ade4e-1c5f-425e-9173-fe7903f4eeab-kube-api-access-56zst\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9\" (UID: \"6f8ade4e-1c5f-425e-9173-fe7903f4eeab\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9"
Sep 17 19:33:41.697545 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:41.697500    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f8ade4e-1c5f-425e-9173-fe7903f4eeab-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9\" (UID: \"6f8ade4e-1c5f-425e-9173-fe7903f4eeab\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9"
Sep 17 19:33:41.700159 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:41.700132    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f8ade4e-1c5f-425e-9173-fe7903f4eeab-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9\" (UID: \"6f8ade4e-1c5f-425e-9173-fe7903f4eeab\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9"
Sep 17 19:33:41.706574 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:41.706552    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56zst\" (UniqueName: \"kubernetes.io/projected/6f8ade4e-1c5f-425e-9173-fe7903f4eeab-kube-api-access-56zst\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9\" (UID: \"6f8ade4e-1c5f-425e-9173-fe7903f4eeab\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9"
Sep 17 19:33:41.862069 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:41.862033    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9"
Sep 17 19:33:42.132980 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:42.132956    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9"]
Sep 17 19:33:42.135061 ip-10-0-136-132 kubenswrapper[2588]: W0917 19:33:42.135025    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f8ade4e_1c5f_425e_9173_fe7903f4eeab.slice/crio-f24fced8bee01756349d83bd19c263a39f700431cd88f767737060f98fa249e2 WatchSource:0}: Error finding container f24fced8bee01756349d83bd19c263a39f700431cd88f767737060f98fa249e2: Status 404 returned error can't find the container with id f24fced8bee01756349d83bd19c263a39f700431cd88f767737060f98fa249e2
Sep 17 19:33:43.104903 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:43.104856    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9" event={"ID":"6f8ade4e-1c5f-425e-9173-fe7903f4eeab","Type":"ContainerStarted","Data":"b167c55d11e71daba41f9b77e2d67243242e36bd5664071c913c6cc66aef448d"}
Sep 17 19:33:43.105424 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:43.104933    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9" event={"ID":"6f8ade4e-1c5f-425e-9173-fe7903f4eeab","Type":"ContainerStarted","Data":"f24fced8bee01756349d83bd19c263a39f700431cd88f767737060f98fa249e2"}
Sep 17 19:33:43.105725 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:43.105701    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9"
Sep 17 19:33:43.125168 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:43.125098    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9" podStartSLOduration=2.125079546 podStartE2EDuration="2.125079546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:33:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:33:43.123099219 +0000 UTC m=+3859.974104620" watchObservedRunningTime="2026-09-17 19:33:43.125079546 +0000 UTC m=+3859.976084945"
Sep 17 19:33:43.601415 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:43.601372    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:33:45.603594 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:33:45.603515    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:00.939730 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:00.939696    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:15.118897 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:15.118868    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9"
Sep 17 19:34:16.537172 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:16.537137    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx"]
Sep 17 19:34:16.540560 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:16.540520    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx"
Sep 17 19:34:16.552351 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:16.552325    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx"]
Sep 17 19:34:16.580367 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:16.580336    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caa42955-9b04-4a51-9b7a-766976170c8b-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-447dx\" (UID: \"caa42955-9b04-4a51-9b7a-766976170c8b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx"
Sep 17 19:34:16.580520 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:16.580380    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caa42955-9b04-4a51-9b7a-766976170c8b-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-447dx\" (UID: \"caa42955-9b04-4a51-9b7a-766976170c8b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx"
Sep 17 19:34:16.580520 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:16.580398    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caa42955-9b04-4a51-9b7a-766976170c8b-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-447dx\" (UID: \"caa42955-9b04-4a51-9b7a-766976170c8b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx"
Sep 17 19:34:16.580520 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:16.580488    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv7lg\" (UniqueName: \"kubernetes.io/projected/caa42955-9b04-4a51-9b7a-766976170c8b-kube-api-access-mv7lg\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-447dx\" (UID: \"caa42955-9b04-4a51-9b7a-766976170c8b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx"
Sep 17 19:34:16.580520 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:16.580515    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caa42955-9b04-4a51-9b7a-766976170c8b-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-447dx\" (UID: \"caa42955-9b04-4a51-9b7a-766976170c8b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx"
Sep 17 19:34:16.580715 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:16.580566    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caa42955-9b04-4a51-9b7a-766976170c8b-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-447dx\" (UID: \"caa42955-9b04-4a51-9b7a-766976170c8b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx"
Sep 17 19:34:16.681186 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:16.681149    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caa42955-9b04-4a51-9b7a-766976170c8b-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-447dx\" (UID: \"caa42955-9b04-4a51-9b7a-766976170c8b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx"
Sep 17 19:34:16.681356 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:16.681194    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caa42955-9b04-4a51-9b7a-766976170c8b-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-447dx\" (UID: \"caa42955-9b04-4a51-9b7a-766976170c8b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx"
Sep 17 19:34:16.681356 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:16.681266    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv7lg\" (UniqueName: \"kubernetes.io/projected/caa42955-9b04-4a51-9b7a-766976170c8b-kube-api-access-mv7lg\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-447dx\" (UID: \"caa42955-9b04-4a51-9b7a-766976170c8b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx"
Sep 17 19:34:16.681356 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:16.681330    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caa42955-9b04-4a51-9b7a-766976170c8b-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-447dx\" (UID: \"caa42955-9b04-4a51-9b7a-766976170c8b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx"
Sep 17 19:34:16.681561 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:16.681378    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caa42955-9b04-4a51-9b7a-766976170c8b-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-447dx\" (UID: \"caa42955-9b04-4a51-9b7a-766976170c8b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx"
Sep 17 19:34:16.681561 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:16.681411    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caa42955-9b04-4a51-9b7a-766976170c8b-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-447dx\" (UID: \"caa42955-9b04-4a51-9b7a-766976170c8b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx"
Sep 17 19:34:16.681735 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:16.681714    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caa42955-9b04-4a51-9b7a-766976170c8b-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-447dx\" (UID: \"caa42955-9b04-4a51-9b7a-766976170c8b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx"
Sep 17 19:34:16.681777 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:16.681742    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caa42955-9b04-4a51-9b7a-766976170c8b-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-447dx\" (UID: \"caa42955-9b04-4a51-9b7a-766976170c8b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx"
Sep 17 19:34:16.681777 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:16.681767    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caa42955-9b04-4a51-9b7a-766976170c8b-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-447dx\" (UID: \"caa42955-9b04-4a51-9b7a-766976170c8b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx"
Sep 17 19:34:16.683626 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:16.683609    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caa42955-9b04-4a51-9b7a-766976170c8b-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-447dx\" (UID: \"caa42955-9b04-4a51-9b7a-766976170c8b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx"
Sep 17 19:34:16.683749 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:16.683730    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caa42955-9b04-4a51-9b7a-766976170c8b-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-447dx\" (UID: \"caa42955-9b04-4a51-9b7a-766976170c8b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx"
Sep 17 19:34:16.689479 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:16.689451    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv7lg\" (UniqueName: \"kubernetes.io/projected/caa42955-9b04-4a51-9b7a-766976170c8b-kube-api-access-mv7lg\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-447dx\" (UID: \"caa42955-9b04-4a51-9b7a-766976170c8b\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx"
Sep 17 19:34:16.850810 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:16.850767    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx"
Sep 17 19:34:17.132602 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:17.132506    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx"]
Sep 17 19:34:17.134848 ip-10-0-136-132 kubenswrapper[2588]: W0917 19:34:17.134819    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaa42955_9b04_4a51_9b7a_766976170c8b.slice/crio-867218e036cafc9d2d041754235ac035d76c1332fc19869ca072be567b35ec9e WatchSource:0}: Error finding container 867218e036cafc9d2d041754235ac035d76c1332fc19869ca072be567b35ec9e: Status 404 returned error can't find the container with id 867218e036cafc9d2d041754235ac035d76c1332fc19869ca072be567b35ec9e
Sep 17 19:34:17.252270 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:17.252227    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx" event={"ID":"caa42955-9b04-4a51-9b7a-766976170c8b","Type":"ContainerStarted","Data":"037139de044506a5f12d6db1d3188defb28e54a8ca0a6d1bde989b34a49353fd"}
Sep 17 19:34:17.252431 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:17.252315    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx" event={"ID":"caa42955-9b04-4a51-9b7a-766976170c8b","Type":"ContainerStarted","Data":"867218e036cafc9d2d041754235ac035d76c1332fc19869ca072be567b35ec9e"}
Sep 17 19:34:17.274499 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:17.274451    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx" podStartSLOduration=1.2744364 podStartE2EDuration="1.2744364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:34:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:34:17.272625921 +0000 UTC m=+3894.123631319" watchObservedRunningTime="2026-09-17 19:34:17.2744364 +0000 UTC m=+3894.125441799"
Sep 17 19:34:17.851183 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:17.851145    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx"
Sep 17 19:34:25.775515 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:25.775484    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xlklb_07ee31e0-43bb-4940-837d-969bd840492b/console-operator/1.log"
Sep 17 19:34:25.780742 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:25.780721    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovn-acl-logging/0.log"
Sep 17 19:34:25.782318 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:25.782298    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xlklb_07ee31e0-43bb-4940-837d-969bd840492b/console-operator/1.log"
Sep 17 19:34:25.787268 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:25.787251    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovn-acl-logging/0.log"
Sep 17 19:34:26.286474 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:26.286431    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx"]
Sep 17 19:34:26.288162 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:26.288129    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9"]
Sep 17 19:34:26.297233 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:34:26.297198    2588 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:34:26.297233 ip-10-0-136-132 kubenswrapper[2588]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:34:26.297233 ip-10-0-136-132 kubenswrapper[2588]:         , stdout: , stderr: , exit code -1
Sep 17 19:34:26.297233 ip-10-0-136-132 kubenswrapper[2588]:  > containerID="037139de044506a5f12d6db1d3188defb28e54a8ca0a6d1bde989b34a49353fd" cmd=["/bin/sleep","15"]
Sep 17 19:34:26.297490 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:26.297246    2588 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx" message="" err=<
Sep 17 19:34:26.297490 ip-10-0-136-132 kubenswrapper[2588]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:34:26.297490 ip-10-0-136-132 kubenswrapper[2588]:         , stdout: , stderr: , exit code -1
Sep 17 19:34:26.297490 ip-10-0-136-132 kubenswrapper[2588]:  >
Sep 17 19:34:26.297490 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:34:26.297259    2588 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:34:26.297490 ip-10-0-136-132 kubenswrapper[2588]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:34:26.297490 ip-10-0-136-132 kubenswrapper[2588]:         , stdout: , stderr: , exit code -1
Sep 17 19:34:26.297490 ip-10-0-136-132 kubenswrapper[2588]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx" podUID="caa42955-9b04-4a51-9b7a-766976170c8b" containerName="main" containerID="cri-o://037139de044506a5f12d6db1d3188defb28e54a8ca0a6d1bde989b34a49353fd"
Sep 17 19:34:26.297490 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:26.297289    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx" podUID="caa42955-9b04-4a51-9b7a-766976170c8b" containerName="main" containerID="cri-o://037139de044506a5f12d6db1d3188defb28e54a8ca0a6d1bde989b34a49353fd" gracePeriod=60
Sep 17 19:34:26.318118 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:26.318076    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:26.413678 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:26.413647    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:26.556873 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:26.556846    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx"
Sep 17 19:34:26.644467 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:26.644427    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mv7lg\" (UniqueName: \"kubernetes.io/projected/caa42955-9b04-4a51-9b7a-766976170c8b-kube-api-access-mv7lg\") pod \"caa42955-9b04-4a51-9b7a-766976170c8b\" (UID: \"caa42955-9b04-4a51-9b7a-766976170c8b\") "
Sep 17 19:34:26.644677 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:26.644478    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caa42955-9b04-4a51-9b7a-766976170c8b-home\") pod \"caa42955-9b04-4a51-9b7a-766976170c8b\" (UID: \"caa42955-9b04-4a51-9b7a-766976170c8b\") "
Sep 17 19:34:26.644677 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:26.644513    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caa42955-9b04-4a51-9b7a-766976170c8b-model-cache\") pod \"caa42955-9b04-4a51-9b7a-766976170c8b\" (UID: \"caa42955-9b04-4a51-9b7a-766976170c8b\") "
Sep 17 19:34:26.644677 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:26.644570    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caa42955-9b04-4a51-9b7a-766976170c8b-tls-certs\") pod \"caa42955-9b04-4a51-9b7a-766976170c8b\" (UID: \"caa42955-9b04-4a51-9b7a-766976170c8b\") "
Sep 17 19:34:26.644677 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:26.644602    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caa42955-9b04-4a51-9b7a-766976170c8b-dshm\") pod \"caa42955-9b04-4a51-9b7a-766976170c8b\" (UID: \"caa42955-9b04-4a51-9b7a-766976170c8b\") "
Sep 17 19:34:26.644911 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:26.644804    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa42955-9b04-4a51-9b7a-766976170c8b-model-cache" (OuterVolumeSpecName: "model-cache") pod "caa42955-9b04-4a51-9b7a-766976170c8b" (UID: "caa42955-9b04-4a51-9b7a-766976170c8b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:34:26.644911 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:26.644819    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa42955-9b04-4a51-9b7a-766976170c8b-home" (OuterVolumeSpecName: "home") pod "caa42955-9b04-4a51-9b7a-766976170c8b" (UID: "caa42955-9b04-4a51-9b7a-766976170c8b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:34:26.646930 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:26.646899    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa42955-9b04-4a51-9b7a-766976170c8b-dshm" (OuterVolumeSpecName: "dshm") pod "caa42955-9b04-4a51-9b7a-766976170c8b" (UID: "caa42955-9b04-4a51-9b7a-766976170c8b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:34:26.647057 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:26.646939    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caa42955-9b04-4a51-9b7a-766976170c8b-kube-api-access-mv7lg" (OuterVolumeSpecName: "kube-api-access-mv7lg") pod "caa42955-9b04-4a51-9b7a-766976170c8b" (UID: "caa42955-9b04-4a51-9b7a-766976170c8b"). InnerVolumeSpecName "kube-api-access-mv7lg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:34:26.647057 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:26.646986    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caa42955-9b04-4a51-9b7a-766976170c8b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "caa42955-9b04-4a51-9b7a-766976170c8b" (UID: "caa42955-9b04-4a51-9b7a-766976170c8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:34:26.745083 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:26.745049    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caa42955-9b04-4a51-9b7a-766976170c8b-tmp-dir\") pod \"caa42955-9b04-4a51-9b7a-766976170c8b\" (UID: \"caa42955-9b04-4a51-9b7a-766976170c8b\") "
Sep 17 19:34:26.745275 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:26.745156    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv7lg\" (UniqueName: \"kubernetes.io/projected/caa42955-9b04-4a51-9b7a-766976170c8b-kube-api-access-mv7lg\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:34:26.745275 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:26.745168    2588 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caa42955-9b04-4a51-9b7a-766976170c8b-home\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:34:26.745275 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:26.745177    2588 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caa42955-9b04-4a51-9b7a-766976170c8b-model-cache\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:34:26.745275 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:26.745186    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caa42955-9b04-4a51-9b7a-766976170c8b-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:34:26.745275 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:26.745193    2588 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caa42955-9b04-4a51-9b7a-766976170c8b-dshm\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:34:26.745497 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:26.745358    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa42955-9b04-4a51-9b7a-766976170c8b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "caa42955-9b04-4a51-9b7a-766976170c8b" (UID: "caa42955-9b04-4a51-9b7a-766976170c8b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:34:26.845503 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:26.845460    2588 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caa42955-9b04-4a51-9b7a-766976170c8b-tmp-dir\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:34:27.294080 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:27.293987    2588 generic.go:356] "Generic (PLEG): container finished" podID="caa42955-9b04-4a51-9b7a-766976170c8b" containerID="037139de044506a5f12d6db1d3188defb28e54a8ca0a6d1bde989b34a49353fd" exitCode=0
Sep 17 19:34:27.294080 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:27.294040    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx"
Sep 17 19:34:27.294280 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:27.294074    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx" event={"ID":"caa42955-9b04-4a51-9b7a-766976170c8b","Type":"ContainerDied","Data":"037139de044506a5f12d6db1d3188defb28e54a8ca0a6d1bde989b34a49353fd"}
Sep 17 19:34:27.294280 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:27.294133    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx" event={"ID":"caa42955-9b04-4a51-9b7a-766976170c8b","Type":"ContainerDied","Data":"867218e036cafc9d2d041754235ac035d76c1332fc19869ca072be567b35ec9e"}
Sep 17 19:34:27.294280 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:27.294163    2588 scope.go:117] "RemoveContainer" containerID="037139de044506a5f12d6db1d3188defb28e54a8ca0a6d1bde989b34a49353fd"
Sep 17 19:34:27.303347 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:27.303158    2588 scope.go:117] "RemoveContainer" containerID="037139de044506a5f12d6db1d3188defb28e54a8ca0a6d1bde989b34a49353fd"
Sep 17 19:34:27.303502 ip-10-0-136-132 kubenswrapper[2588]: E0917 19:34:27.303449    2588 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"037139de044506a5f12d6db1d3188defb28e54a8ca0a6d1bde989b34a49353fd\": container with ID starting with 037139de044506a5f12d6db1d3188defb28e54a8ca0a6d1bde989b34a49353fd not found: ID does not exist" containerID="037139de044506a5f12d6db1d3188defb28e54a8ca0a6d1bde989b34a49353fd"
Sep 17 19:34:27.303566 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:27.303510    2588 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"037139de044506a5f12d6db1d3188defb28e54a8ca0a6d1bde989b34a49353fd"} err="failed to get container status \"037139de044506a5f12d6db1d3188defb28e54a8ca0a6d1bde989b34a49353fd\": rpc error: code = NotFound desc = could not find container \"037139de044506a5f12d6db1d3188defb28e54a8ca0a6d1bde989b34a49353fd\": container with ID starting with 037139de044506a5f12d6db1d3188defb28e54a8ca0a6d1bde989b34a49353fd not found: ID does not exist"
Sep 17 19:34:27.315126 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:27.315102    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx"]
Sep 17 19:34:27.321339 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:27.321316    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-447dx"]
Sep 17 19:34:27.506247 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:27.506209    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caa42955-9b04-4a51-9b7a-766976170c8b" path="/var/lib/kubelet/pods/caa42955-9b04-4a51-9b7a-766976170c8b/volumes"
Sep 17 19:34:29.553300 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:29.553267    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:35.119170 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:35.119136    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9" podUID="6f8ade4e-1c5f-425e-9173-fe7903f4eeab" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:34:41.307757 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:41.307710    2588 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9" podUID="6f8ade4e-1c5f-425e-9173-fe7903f4eeab" containerName="main" containerID="cri-o://b167c55d11e71daba41f9b77e2d67243242e36bd5664071c913c6cc66aef448d" gracePeriod=45
Sep 17 19:34:45.119203 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:45.119164    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9" podUID="6f8ade4e-1c5f-425e-9173-fe7903f4eeab" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:34:55.118720 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:55.118679    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9" podUID="6f8ade4e-1c5f-425e-9173-fe7903f4eeab" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:34:55.119111 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:34:55.118782    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9"
Sep 17 19:35:05.118823 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:05.118783    2588 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9" podUID="6f8ade4e-1c5f-425e-9173-fe7903f4eeab" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:35:11.474701 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:11.474338    2588 generic.go:356] "Generic (PLEG): container finished" podID="6f8ade4e-1c5f-425e-9173-fe7903f4eeab" containerID="b167c55d11e71daba41f9b77e2d67243242e36bd5664071c913c6cc66aef448d" exitCode=0
Sep 17 19:35:11.474701 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:11.474458    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9" event={"ID":"6f8ade4e-1c5f-425e-9173-fe7903f4eeab","Type":"ContainerDied","Data":"b167c55d11e71daba41f9b77e2d67243242e36bd5664071c913c6cc66aef448d"}
Sep 17 19:35:11.552430 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:11.552402    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9"
Sep 17 19:35:11.710618 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:11.710497    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f8ade4e-1c5f-425e-9173-fe7903f4eeab-tls-certs\") pod \"6f8ade4e-1c5f-425e-9173-fe7903f4eeab\" (UID: \"6f8ade4e-1c5f-425e-9173-fe7903f4eeab\") "
Sep 17 19:35:11.710618 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:11.710571    2588 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56zst\" (UniqueName: \"kubernetes.io/projected/6f8ade4e-1c5f-425e-9173-fe7903f4eeab-kube-api-access-56zst\") pod \"6f8ade4e-1c5f-425e-9173-fe7903f4eeab\" (UID: \"6f8ade4e-1c5f-425e-9173-fe7903f4eeab\") "
Sep 17 19:35:11.712688 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:11.712657    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f8ade4e-1c5f-425e-9173-fe7903f4eeab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6f8ade4e-1c5f-425e-9173-fe7903f4eeab" (UID: "6f8ade4e-1c5f-425e-9173-fe7903f4eeab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:35:11.712688 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:11.712668    2588 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f8ade4e-1c5f-425e-9173-fe7903f4eeab-kube-api-access-56zst" (OuterVolumeSpecName: "kube-api-access-56zst") pod "6f8ade4e-1c5f-425e-9173-fe7903f4eeab" (UID: "6f8ade4e-1c5f-425e-9173-fe7903f4eeab"). InnerVolumeSpecName "kube-api-access-56zst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:35:11.811191 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:11.811152    2588 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f8ade4e-1c5f-425e-9173-fe7903f4eeab-tls-certs\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:35:11.811191 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:11.811185    2588 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56zst\" (UniqueName: \"kubernetes.io/projected/6f8ade4e-1c5f-425e-9173-fe7903f4eeab-kube-api-access-56zst\") on node \"ip-10-0-136-132.ec2.internal\" DevicePath \"\""
Sep 17 19:35:12.482391 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:12.482360    2588 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9"
Sep 17 19:35:12.482391 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:12.482382    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9" event={"ID":"6f8ade4e-1c5f-425e-9173-fe7903f4eeab","Type":"ContainerDied","Data":"f24fced8bee01756349d83bd19c263a39f700431cd88f767737060f98fa249e2"}
Sep 17 19:35:12.482970 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:12.482438    2588 scope.go:117] "RemoveContainer" containerID="b167c55d11e71daba41f9b77e2d67243242e36bd5664071c913c6cc66aef448d"
Sep 17 19:35:12.504371 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:12.504343    2588 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9"]
Sep 17 19:35:12.508465 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:12.508435    2588 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-cr4j9"]
Sep 17 19:35:13.506277 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:13.506245    2588 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f8ade4e-1c5f-425e-9173-fe7903f4eeab" path="/var/lib/kubelet/pods/6f8ade4e-1c5f-425e-9173-fe7903f4eeab/volumes"
Sep 17 19:35:13.517168 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:13.517141    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:35:15.048609 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:15.048574    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:35:15.137816 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:15.137753    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:35:19.474938 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:19.474903    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:35:25.025318 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:25.025263    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:35:25.124817 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:25.124779    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:35:38.138545 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:38.138490    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-72h4k_585c7370-1cb3-4035-ac56-be5c3f566027/istio-proxy/0.log"
Sep 17 19:35:38.140953 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:38.140929    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-72h4k_585c7370-1cb3-4035-ac56-be5c3f566027/istio-proxy/1.log"
Sep 17 19:35:39.159508 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:39.159469    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-72h4k_585c7370-1cb3-4035-ac56-be5c3f566027/istio-proxy/0.log"
Sep 17 19:35:39.162406 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:39.162380    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-72h4k_585c7370-1cb3-4035-ac56-be5c3f566027/istio-proxy/1.log"
Sep 17 19:35:40.002408 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:40.002364    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-6hlmm_067e3a44-769a-4ad5-87fb-e25fd9430437/authorino/0.log"
Sep 17 19:35:45.737823 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:45.737790    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:35:45.889441 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:45.889410    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:35:45.980444 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:45.980414    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:35:48.988008 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:48.987976    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fs688_c554cd64-2695-47d3-999a-2d8bf8d7ef63/global-pull-secret-syncer/0.log"
Sep 17 19:35:49.032722 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:49.032677    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log"
Sep 17 19:35:49.195247 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:49.195221    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-132.ec2.internal_189465409bf7b6f4cb62d3daabaf12f1/haproxy/0.log"
Sep 17 19:35:51.818440 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:51.818404    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:35:52.175451 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:52.175364    2588 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2t758_10d63418-937e-4dee-a71f-592cfdff55ca/konnectivity-agent/0.log" limit=1024
Sep 17 19:35:53.677328 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:53.677297    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-6hlmm_067e3a44-769a-4ad5-87fb-e25fd9430437/authorino/0.log"
Sep 17 19:35:55.643931 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:55.643902    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-j5s56_49a517a5-36f1-4482-b8fc-fa78fb902040/cluster-monitoring-operator/0.log"
Sep 17 19:35:55.811498 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:55.811468    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cfccv_191b6fe8-f5c2-4a23-86b7-5b3f96adf26d/node-exporter/0.log"
Sep 17 19:35:55.842288 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:55.842252    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cfccv_191b6fe8-f5c2-4a23-86b7-5b3f96adf26d/kube-rbac-proxy/0.log"
Sep 17 19:35:55.866988 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:55.866965    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cfccv_191b6fe8-f5c2-4a23-86b7-5b3f96adf26d/init-textfile/0.log"
Sep 17 19:35:58.539515 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:58.539488    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xlklb_07ee31e0-43bb-4940-837d-969bd840492b/console-operator/1.log"
Sep 17 19:35:58.544439 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:58.544409    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-xlklb_07ee31e0-43bb-4940-837d-969bd840492b/console-operator/2.log"
Sep 17 19:35:59.344622 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:59.344585    2588 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4lb2x/perf-node-gather-daemonset-fpb2r"]
Sep 17 19:35:59.344894 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:59.344881    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caa42955-9b04-4a51-9b7a-766976170c8b" containerName="main"
Sep 17 19:35:59.344894 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:59.344895    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="caa42955-9b04-4a51-9b7a-766976170c8b" containerName="main"
Sep 17 19:35:59.344991 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:59.344910    2588 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f8ade4e-1c5f-425e-9173-fe7903f4eeab" containerName="main"
Sep 17 19:35:59.344991 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:59.344915    2588 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f8ade4e-1c5f-425e-9173-fe7903f4eeab" containerName="main"
Sep 17 19:35:59.344991 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:59.344968    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f8ade4e-1c5f-425e-9173-fe7903f4eeab" containerName="main"
Sep 17 19:35:59.344991 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:59.344976    2588 memory_manager.go:368] "RemoveStaleState removing state" podUID="caa42955-9b04-4a51-9b7a-766976170c8b" containerName="main"
Sep 17 19:35:59.347921 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:59.347903    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-fpb2r"
Sep 17 19:35:59.351449 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:59.351420    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4lb2x\"/\"kube-root-ca.crt\""
Sep 17 19:35:59.351634 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:59.351614    2588 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4lb2x\"/\"default-dockercfg-zzdfx\""
Sep 17 19:35:59.352331 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:59.352312    2588 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4lb2x\"/\"openshift-service-ca.crt\""
Sep 17 19:35:59.368948 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:59.368923    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4lb2x/perf-node-gather-daemonset-fpb2r"]
Sep 17 19:35:59.466202 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:59.466161    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/89e9bc88-e7b8-45b2-8bbd-0bd6cbfb72c9-lib-modules\") pod \"perf-node-gather-daemonset-fpb2r\" (UID: \"89e9bc88-e7b8-45b2-8bbd-0bd6cbfb72c9\") " pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-fpb2r"
Sep 17 19:35:59.466202 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:59.466201    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxrj2\" (UniqueName: \"kubernetes.io/projected/89e9bc88-e7b8-45b2-8bbd-0bd6cbfb72c9-kube-api-access-wxrj2\") pod \"perf-node-gather-daemonset-fpb2r\" (UID: \"89e9bc88-e7b8-45b2-8bbd-0bd6cbfb72c9\") " pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-fpb2r"
Sep 17 19:35:59.466468 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:59.466231    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/89e9bc88-e7b8-45b2-8bbd-0bd6cbfb72c9-podres\") pod \"perf-node-gather-daemonset-fpb2r\" (UID: \"89e9bc88-e7b8-45b2-8bbd-0bd6cbfb72c9\") " pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-fpb2r"
Sep 17 19:35:59.466468 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:59.466322    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89e9bc88-e7b8-45b2-8bbd-0bd6cbfb72c9-sys\") pod \"perf-node-gather-daemonset-fpb2r\" (UID: \"89e9bc88-e7b8-45b2-8bbd-0bd6cbfb72c9\") " pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-fpb2r"
Sep 17 19:35:59.466468 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:59.466341    2588 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/89e9bc88-e7b8-45b2-8bbd-0bd6cbfb72c9-proc\") pod \"perf-node-gather-daemonset-fpb2r\" (UID: \"89e9bc88-e7b8-45b2-8bbd-0bd6cbfb72c9\") " pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-fpb2r"
Sep 17 19:35:59.566871 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:59.566833    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/89e9bc88-e7b8-45b2-8bbd-0bd6cbfb72c9-lib-modules\") pod \"perf-node-gather-daemonset-fpb2r\" (UID: \"89e9bc88-e7b8-45b2-8bbd-0bd6cbfb72c9\") " pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-fpb2r"
Sep 17 19:35:59.567337 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:59.566872    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxrj2\" (UniqueName: \"kubernetes.io/projected/89e9bc88-e7b8-45b2-8bbd-0bd6cbfb72c9-kube-api-access-wxrj2\") pod \"perf-node-gather-daemonset-fpb2r\" (UID: \"89e9bc88-e7b8-45b2-8bbd-0bd6cbfb72c9\") " pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-fpb2r"
Sep 17 19:35:59.567337 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:59.566996    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/89e9bc88-e7b8-45b2-8bbd-0bd6cbfb72c9-podres\") pod \"perf-node-gather-daemonset-fpb2r\" (UID: \"89e9bc88-e7b8-45b2-8bbd-0bd6cbfb72c9\") " pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-fpb2r"
Sep 17 19:35:59.567337 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:59.567060    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/89e9bc88-e7b8-45b2-8bbd-0bd6cbfb72c9-lib-modules\") pod \"perf-node-gather-daemonset-fpb2r\" (UID: \"89e9bc88-e7b8-45b2-8bbd-0bd6cbfb72c9\") " pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-fpb2r"
Sep 17 19:35:59.567337 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:59.567097    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89e9bc88-e7b8-45b2-8bbd-0bd6cbfb72c9-sys\") pod \"perf-node-gather-daemonset-fpb2r\" (UID: \"89e9bc88-e7b8-45b2-8bbd-0bd6cbfb72c9\") " pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-fpb2r"
Sep 17 19:35:59.567337 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:59.567116    2588 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/89e9bc88-e7b8-45b2-8bbd-0bd6cbfb72c9-proc\") pod \"perf-node-gather-daemonset-fpb2r\" (UID: \"89e9bc88-e7b8-45b2-8bbd-0bd6cbfb72c9\") " pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-fpb2r"
Sep 17 19:35:59.567337 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:59.567144    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/89e9bc88-e7b8-45b2-8bbd-0bd6cbfb72c9-podres\") pod \"perf-node-gather-daemonset-fpb2r\" (UID: \"89e9bc88-e7b8-45b2-8bbd-0bd6cbfb72c9\") " pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-fpb2r"
Sep 17 19:35:59.567337 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:59.567185    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/89e9bc88-e7b8-45b2-8bbd-0bd6cbfb72c9-sys\") pod \"perf-node-gather-daemonset-fpb2r\" (UID: \"89e9bc88-e7b8-45b2-8bbd-0bd6cbfb72c9\") " pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-fpb2r"
Sep 17 19:35:59.567337 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:59.567205    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/89e9bc88-e7b8-45b2-8bbd-0bd6cbfb72c9-proc\") pod \"perf-node-gather-daemonset-fpb2r\" (UID: \"89e9bc88-e7b8-45b2-8bbd-0bd6cbfb72c9\") " pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-fpb2r"
Sep 17 19:35:59.576520 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:59.576496    2588 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxrj2\" (UniqueName: \"kubernetes.io/projected/89e9bc88-e7b8-45b2-8bbd-0bd6cbfb72c9-kube-api-access-wxrj2\") pod \"perf-node-gather-daemonset-fpb2r\" (UID: \"89e9bc88-e7b8-45b2-8bbd-0bd6cbfb72c9\") " pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-fpb2r"
Sep 17 19:35:59.657718 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:59.657628    2588 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-fpb2r"
Sep 17 19:35:59.680200 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:59.680177    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-x69bl_96541b99-da29-416c-916f-91fbb7207ef1/volume-data-source-validator/0.log"
Sep 17 19:35:59.928864 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:35:59.928837    2588 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4lb2x/perf-node-gather-daemonset-fpb2r"]
Sep 17 19:35:59.931332 ip-10-0-136-132 kubenswrapper[2588]: W0917 19:35:59.931305    2588 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod89e9bc88_e7b8_45b2_8bbd_0bd6cbfb72c9.slice/crio-63e04ae9f810a8803b910796a97d15006383fcbdafef8402d38a6efbd831cd7a WatchSource:0}: Error finding container 63e04ae9f810a8803b910796a97d15006383fcbdafef8402d38a6efbd831cd7a: Status 404 returned error can't find the container with id 63e04ae9f810a8803b910796a97d15006383fcbdafef8402d38a6efbd831cd7a
Sep 17 19:36:00.681581 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:00.681510    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-fpb2r" event={"ID":"89e9bc88-e7b8-45b2-8bbd-0bd6cbfb72c9","Type":"ContainerStarted","Data":"283814c50c69df3071e72228a1584e570c82a23928693b082106af515457cb97"}
Sep 17 19:36:00.681984 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:00.681604    2588 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-fpb2r" event={"ID":"89e9bc88-e7b8-45b2-8bbd-0bd6cbfb72c9","Type":"ContainerStarted","Data":"63e04ae9f810a8803b910796a97d15006383fcbdafef8402d38a6efbd831cd7a"}
Sep 17 19:36:00.681984 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:00.681632    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-fpb2r"
Sep 17 19:36:00.698239 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:00.698190    2588 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-fpb2r" podStartSLOduration=1.698175441 podStartE2EDuration="1.698175441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:36:00.697134458 +0000 UTC m=+3997.548139857" watchObservedRunningTime="2026-09-17 19:36:00.698175441 +0000 UTC m=+3997.549180837"
Sep 17 19:36:00.734448 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:00.734417    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jq7kv_6a7075b4-9ee1-4495-86e5-b2377d9b1b06/dns/0.log"
Sep 17 19:36:00.756050 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:00.756031    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jq7kv_6a7075b4-9ee1-4495-86e5-b2377d9b1b06/kube-rbac-proxy/0.log"
Sep 17 19:36:00.824711 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:00.824686    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-745b6_61513726-d6fb-4e35-a39f-818bcb1a9596/dns-node-resolver/0.log"
Sep 17 19:36:01.415008 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:01.414970    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5cb86_947781e1-4423-435e-b44f-53ca352f8df7/node-ca/0.log"
Sep 17 19:36:02.531988 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:02.531961    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-72h4k_585c7370-1cb3-4035-ac56-be5c3f566027/istio-proxy/0.log"
Sep 17 19:36:02.538279 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:02.538247    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-72h4k_585c7370-1cb3-4035-ac56-be5c3f566027/istio-proxy/1.log"
Sep 17 19:36:03.106385 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:03.106350    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6llb8_676d0454-70c3-4491-b9bc-7d934002b6be/serve-healthcheck-canary/0.log"
Sep 17 19:36:03.554108 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:03.554080    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-cf8bw_63716ecc-bcb1-44d0-96c4-40ac49599a79/insights-operator/0.log"
Sep 17 19:36:03.555305 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:03.555286    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-cf8bw_63716ecc-bcb1-44d0-96c4-40ac49599a79/insights-operator/1.log"
Sep 17 19:36:03.648155 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:03.648115    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w9vq7_904b068f-aa14-406f-835c-bf966bb4c9ff/kube-rbac-proxy/0.log"
Sep 17 19:36:03.675121 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:03.675094    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w9vq7_904b068f-aa14-406f-835c-bf966bb4c9ff/exporter/0.log"
Sep 17 19:36:03.698017 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:03.697984    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w9vq7_904b068f-aa14-406f-835c-bf966bb4c9ff/extractor/0.log"
Sep 17 19:36:06.694558 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:06.694507    2588 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-fpb2r"
Sep 17 19:36:06.798302 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:06.798272    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7594644fcd-x58df_f72915b1-27d1-41fe-94f6-eac990c3e860/manager/0.log"
Sep 17 19:36:06.827599 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:06.827565    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-vsrhx_85f3da13-01f5-47dc-a03d-7529532edff2/openshift-lws-operator/0.log"
Sep 17 19:36:08.968559 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:08.968517    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-72h4k_585c7370-1cb3-4035-ac56-be5c3f566027/istio-proxy/0.log"
Sep 17 19:36:08.970846 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:08.970828    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-72h4k_585c7370-1cb3-4035-ac56-be5c3f566027/istio-proxy/1.log"
Sep 17 19:36:10.364859 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:10.364815    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-72h4k_585c7370-1cb3-4035-ac56-be5c3f566027/istio-proxy/0.log"
Sep 17 19:36:10.366570 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:10.366540    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-72h4k_585c7370-1cb3-4035-ac56-be5c3f566027/istio-proxy/1.log"
Sep 17 19:36:15.433429 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:15.433396    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kpswk_a7f6c88f-5861-4089-a4a8-75bb85c16c98/kube-multus-additional-cni-plugins/0.log"
Sep 17 19:36:15.471136 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:15.471094    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kpswk_a7f6c88f-5861-4089-a4a8-75bb85c16c98/egress-router-binary-copy/0.log"
Sep 17 19:36:15.507144 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:15.507088    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kpswk_a7f6c88f-5861-4089-a4a8-75bb85c16c98/cni-plugins/0.log"
Sep 17 19:36:15.541170 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:15.541143    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kpswk_a7f6c88f-5861-4089-a4a8-75bb85c16c98/bond-cni-plugin/0.log"
Sep 17 19:36:15.577071 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:15.577045    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kpswk_a7f6c88f-5861-4089-a4a8-75bb85c16c98/routeoverride-cni/0.log"
Sep 17 19:36:15.611081 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:15.611037    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kpswk_a7f6c88f-5861-4089-a4a8-75bb85c16c98/whereabouts-cni-bincopy/0.log"
Sep 17 19:36:15.649163 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:15.649136    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kpswk_a7f6c88f-5861-4089-a4a8-75bb85c16c98/whereabouts-cni/0.log"
Sep 17 19:36:16.118452 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:16.118387    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-f96pg_8462659d-7d75-4d56-b4c1-75dfc4aea577/kube-multus/0.log"
Sep 17 19:36:16.262739 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:16.262705    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bnq8f_1c01ccd8-9e9c-467f-a2af-50720f272038/network-metrics-daemon/0.log"
Sep 17 19:36:16.298243 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:16.298216    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bnq8f_1c01ccd8-9e9c-467f-a2af-50720f272038/kube-rbac-proxy/0.log"
Sep 17 19:36:17.267208 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:17.267181    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovn-controller/0.log"
Sep 17 19:36:17.321830 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:17.321780    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovn-acl-logging/0.log"
Sep 17 19:36:17.321830 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:17.321799    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovn-acl-logging/1.log"
Sep 17 19:36:17.350175 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:17.350149    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/kube-rbac-proxy-node/0.log"
Sep 17 19:36:17.403904 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:17.403877    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 19:36:17.430461 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:17.430436    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/northd/0.log"
Sep 17 19:36:17.456388 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:17.456366    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/nbdb/0.log"
Sep 17 19:36:17.479457 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:17.479427    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/sbdb/0.log"
Sep 17 19:36:17.538057 ip-10-0-136-132 kubenswrapper[2588]: I0917 19:36:17.537977    2588 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2rqhq_c69432b1-9a81-4293-bd1c-4df1df813ba4/ovnkube-controller/0.log"