Sep 16 17:22:55.222787 ip-10-0-136-122 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 17:22:55.222984 ip-10-0-136-122 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 17:22:55.223102 ip-10-0-136-122 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 17:22:55.223691 ip-10-0-136-122 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 17:23:06.613665 ip-10-0-136-122 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 17:23:06.613680 ip-10-0-136-122 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 7eadcae276ed49e697fcdc6feacaf99a --
Sep 16 17:24:38.528141 ip-10-0-136-122 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 17:24:38.891421 ip-10-0-136-122 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 17:24:38.891421 ip-10-0-136-122 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 17:24:38.891421 ip-10-0-136-122 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 17:24:38.891421 ip-10-0-136-122 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 17:24:38.891421 ip-10-0-136-122 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 17:24:38.892239 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.892151    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 17:24:38.899148 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.899110    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 17:24:38.899329 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.899314    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 17:24:38.899397 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.899390    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 17:24:38.899462 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.899456    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 17:24:38.899518 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.899511    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 17:24:38.899580 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.899567    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 17:24:38.899635 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.899629    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 17:24:38.899691 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.899684    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 17:24:38.899750 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.899744    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 17:24:38.899803 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.899798    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 17:24:38.899857 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.899851    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 17:24:38.899938 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.899930    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 17:24:38.899985 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.899962    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 17:24:38.899985 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.899970    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 17:24:38.899985 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.899975    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 17:24:38.899985 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.899979    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 17:24:38.900160 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.899989    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 17:24:38.900160 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.899998    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 17:24:38.900160 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900003    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 17:24:38.900160 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900011    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 17:24:38.900160 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900016    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 17:24:38.900160 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900020    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 17:24:38.900160 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900025    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 17:24:38.900160 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900030    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 17:24:38.900160 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900035    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 17:24:38.900160 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900040    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 17:24:38.900160 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900045    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 17:24:38.900160 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900049    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 17:24:38.900160 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900060    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 17:24:38.900160 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900064    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 17:24:38.900160 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900069    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 17:24:38.900160 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900074    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 17:24:38.900160 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900078    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 17:24:38.900160 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900083    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 17:24:38.900160 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900088    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 17:24:38.900160 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900092    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 17:24:38.900974 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900097    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 17:24:38.900974 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900101    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 17:24:38.900974 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900106    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 17:24:38.900974 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900111    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 17:24:38.900974 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900115    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 17:24:38.900974 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900124    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 17:24:38.900974 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900129    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 17:24:38.900974 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900134    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 17:24:38.900974 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900142    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 17:24:38.900974 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900298    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 17:24:38.900974 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900309    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 17:24:38.900974 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900313    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 17:24:38.900974 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900319    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 17:24:38.900974 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900324    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 17:24:38.900974 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900328    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 17:24:38.900974 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900333    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 17:24:38.900974 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900337    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 17:24:38.900974 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900341    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 17:24:38.900974 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900345    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 17:24:38.900974 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900350    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 17:24:38.901783 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900353    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 17:24:38.901783 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900358    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 17:24:38.901783 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900362    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 17:24:38.901783 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900366    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 17:24:38.901783 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900370    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 17:24:38.901783 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900375    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 17:24:38.901783 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900379    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 17:24:38.901783 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900386    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 17:24:38.901783 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900392    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 17:24:38.901783 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900397    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 17:24:38.901783 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900402    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 17:24:38.901783 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900407    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 17:24:38.901783 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900412    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 17:24:38.901783 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900418    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 17:24:38.901783 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900424    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 17:24:38.901783 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900428    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 17:24:38.901783 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900436    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 17:24:38.901783 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900442    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 17:24:38.901783 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900447    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 17:24:38.902595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900451    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 17:24:38.902595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900456    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 17:24:38.902595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900460    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 17:24:38.902595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900465    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 17:24:38.902595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900470    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 17:24:38.902595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900475    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 17:24:38.902595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900480    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 17:24:38.902595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900485    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 17:24:38.902595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900489    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 17:24:38.902595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900495    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 17:24:38.902595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900500    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 17:24:38.902595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900504    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 17:24:38.902595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900509    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 17:24:38.902595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900514    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 17:24:38.902595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900518    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 17:24:38.902595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900522    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 17:24:38.902595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900527    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 17:24:38.902595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900533    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 17:24:38.902595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900539    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 17:24:38.903454 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900544    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 17:24:38.903454 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900549    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 17:24:38.903454 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900554    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 17:24:38.903454 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900559    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 17:24:38.903454 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.900564    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 17:24:38.903454 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901743    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 17:24:38.903454 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901754    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 17:24:38.903454 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901760    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 17:24:38.903454 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901765    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 17:24:38.903454 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901769    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 17:24:38.903454 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901776    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 17:24:38.903454 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901782    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 17:24:38.903454 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901787    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 17:24:38.903454 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901792    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 17:24:38.903454 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901796    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 17:24:38.903454 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901800    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 17:24:38.903454 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901804    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 17:24:38.903454 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901808    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 17:24:38.903454 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901814    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 17:24:38.904299 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901819    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 17:24:38.904299 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901824    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 17:24:38.904299 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901829    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 17:24:38.904299 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901835    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 17:24:38.904299 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901840    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 17:24:38.904299 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901844    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 17:24:38.904299 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901849    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 17:24:38.904299 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901853    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 17:24:38.904299 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901883    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 17:24:38.904299 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901892    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 17:24:38.904299 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901896    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 17:24:38.904299 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901901    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 17:24:38.904299 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901905    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 17:24:38.904299 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901909    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 17:24:38.904299 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901913    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 17:24:38.904299 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901917    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 17:24:38.904299 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901921    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 17:24:38.904299 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901925    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 17:24:38.904299 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901929    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 17:24:38.905115 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901934    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 17:24:38.905115 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901939    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 17:24:38.905115 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901944    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 17:24:38.905115 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901948    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 17:24:38.905115 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901952    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 17:24:38.905115 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901956    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 17:24:38.905115 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901961    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 17:24:38.905115 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901965    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 17:24:38.905115 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901969    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 17:24:38.905115 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901973    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 17:24:38.905115 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901978    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 17:24:38.905115 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901982    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 17:24:38.905115 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901986    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 17:24:38.905115 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901990    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 17:24:38.905115 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901994    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 17:24:38.905115 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.901998    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 17:24:38.905115 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902003    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 17:24:38.905115 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902010    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 17:24:38.905115 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902015    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 17:24:38.905115 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902019    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 17:24:38.905607 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902023    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 17:24:38.905607 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902029    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 17:24:38.905607 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902033    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 17:24:38.905607 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902037    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 17:24:38.905607 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902041    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 17:24:38.905607 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902046    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 17:24:38.905607 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902050    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 17:24:38.905607 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902054    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 17:24:38.905607 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902058    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 17:24:38.905607 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902063    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 17:24:38.905607 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902067    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 17:24:38.905607 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902071    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 17:24:38.905607 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902075    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 17:24:38.905607 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902080    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 17:24:38.905607 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902085    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 17:24:38.905607 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902089    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 17:24:38.905607 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902093    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 17:24:38.905607 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902098    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 17:24:38.905607 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902102    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 17:24:38.905607 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902106    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 17:24:38.906201 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902111    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 17:24:38.906201 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902115    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 17:24:38.906201 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902119    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 17:24:38.906201 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902123    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 17:24:38.906201 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902127    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 17:24:38.906201 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902132    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 17:24:38.906201 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902136    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 17:24:38.906201 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902140    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 17:24:38.906201 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902144    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 17:24:38.906201 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902148    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 17:24:38.906201 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902153    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 17:24:38.906201 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902158    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 17:24:38.906201 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902163    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 17:24:38.906201 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902168    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 17:24:38.906201 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902172    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 17:24:38.906201 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902177    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 17:24:38.906201 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902181    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 17:24:38.906201 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902186    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 17:24:38.906201 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902190    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 17:24:38.906201 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902194    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 17:24:38.906719 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902198    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 17:24:38.906719 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902202    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 17:24:38.906719 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902206    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 17:24:38.906719 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902211    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 17:24:38.906719 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902215    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 17:24:38.906719 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902220    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 17:24:38.906719 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902855    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 17:24:38.906719 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902878    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 17:24:38.906719 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902884    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 17:24:38.906719 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902889    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 17:24:38.906719 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902893    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 17:24:38.906719 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902897    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 17:24:38.906719 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902901    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 17:24:38.906719 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902906    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 17:24:38.906719 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902911    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 17:24:38.906719 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902917    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 17:24:38.906719 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902921    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 17:24:38.906719 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902925    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 17:24:38.906719 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902929    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 17:24:38.906719 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902934    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 17:24:38.907392 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902938    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 17:24:38.907392 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902942    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 17:24:38.907392 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902948    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 17:24:38.907392 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902952    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 17:24:38.907392 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902956    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 17:24:38.907392 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902960    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 17:24:38.907392 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902965    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 17:24:38.907392 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902969    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 17:24:38.907392 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902973    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 17:24:38.907392 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902978    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 17:24:38.907392 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902982    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 17:24:38.907392 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902986    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 17:24:38.907392 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902990    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 17:24:38.907392 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902995    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 17:24:38.907392 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.902999    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 17:24:38.907392 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903003    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 17:24:38.907392 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903007    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 17:24:38.907392 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903013    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 17:24:38.907392 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903017    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 17:24:38.907392 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903022    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 17:24:38.907942 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903027    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 17:24:38.907942 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903031    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 17:24:38.907942 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903035    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 17:24:38.907942 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903041    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 17:24:38.907942 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903045    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 17:24:38.907942 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903049    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 17:24:38.907942 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903053    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 17:24:38.907942 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903058    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 17:24:38.907942 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903063    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 17:24:38.907942 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903067    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 17:24:38.907942 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903071    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 17:24:38.907942 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903075    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 17:24:38.907942 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903079    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 17:24:38.907942 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903084    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 17:24:38.907942 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903088    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 17:24:38.907942 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903093    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 17:24:38.907942 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903098    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 17:24:38.907942 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903102    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 17:24:38.907942 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903106    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 17:24:38.907942 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903110    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 17:24:38.908422 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903114    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 17:24:38.908422 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903118    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 17:24:38.908422 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903123    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 17:24:38.908422 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903127    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 17:24:38.908422 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903131    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 17:24:38.908422 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903136    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 17:24:38.908422 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903140    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 17:24:38.908422 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903144    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 17:24:38.908422 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903149    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 17:24:38.908422 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903154    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 17:24:38.908422 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903159    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 17:24:38.908422 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903163    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 17:24:38.908422 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903167    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 17:24:38.908422 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903171    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 17:24:38.908422 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903175    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 17:24:38.908422 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903180    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 17:24:38.908422 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903184    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 17:24:38.908422 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903188    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 17:24:38.908422 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903192    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 17:24:38.909051 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903196    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 17:24:38.909051 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903203    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 17:24:38.909051 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903209    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 17:24:38.909051 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903213    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 17:24:38.909051 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903218    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 17:24:38.909051 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903222    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 17:24:38.909051 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903226    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 17:24:38.909051 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903231    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 17:24:38.909051 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903235    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 17:24:38.909051 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903240    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 17:24:38.909051 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903244    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 17:24:38.909051 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903248    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 17:24:38.909051 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903253    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 17:24:38.909051 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903257    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 17:24:38.909051 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903263    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 17:24:38.909051 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903267    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 17:24:38.909051 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903272    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 17:24:38.909051 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903276    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 17:24:38.909051 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903280    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 17:24:38.909051 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903284    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 17:24:38.909542 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903289    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 17:24:38.909542 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903293    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 17:24:38.909542 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903297    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 17:24:38.909542 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903302    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 17:24:38.909542 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903309    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 17:24:38.909542 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.903315    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 17:24:38.909542 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903424    2577 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 17:24:38.909542 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903473    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 17:24:38.909542 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903491    2577 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 17:24:38.909542 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903498    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 17:24:38.909542 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903505    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 17:24:38.909542 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903510    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 17:24:38.909542 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903516    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 17:24:38.909542 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903523    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 17:24:38.909542 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903528    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 17:24:38.909542 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903533    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 17:24:38.909542 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903538    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 17:24:38.909542 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903544    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 17:24:38.909542 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903549    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 17:24:38.909542 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903554    2577 flags.go:64] FLAG: --cgroup-root=""
Sep 16 17:24:38.909542 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903559    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 17:24:38.909542 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903564    2577 flags.go:64] FLAG: --client-ca-file=""
Sep 16 17:24:38.910130 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903570    2577 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 17:24:38.910130 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903576    2577 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 17:24:38.910130 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903583    2577 flags.go:64] FLAG: --cluster-domain=""
Sep 16 17:24:38.910130 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903587    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 17:24:38.910130 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903593    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 17:24:38.910130 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903598    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 17:24:38.910130 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903603    2577 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 17:24:38.910130 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903609    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 17:24:38.910130 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903614    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 17:24:38.910130 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903619    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 17:24:38.910130 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903625    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 17:24:38.910130 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903629    2577 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 17:24:38.910130 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903634    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 17:24:38.910130 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903639    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 17:24:38.910130 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903644    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 17:24:38.910130 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903649    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 17:24:38.910130 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903655    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 17:24:38.910130 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903660    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 17:24:38.910130 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903667    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 17:24:38.910130 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903672    2577 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 17:24:38.910130 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903677    2577 flags.go:64] FLAG: --enable-server="true"
Sep 16 17:24:38.910130 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903681    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 17:24:38.910130 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903689    2577 flags.go:64] FLAG: --event-burst="100"
Sep 16 17:24:38.910130 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903694    2577 flags.go:64] FLAG: --event-qps="50"
Sep 16 17:24:38.910130 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903698    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 17:24:38.910737 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903703    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 17:24:38.910737 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903708    2577 flags.go:64] FLAG: --eviction-hard=""
Sep 16 17:24:38.910737 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903714    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 17:24:38.910737 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903719    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 17:24:38.910737 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903724    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 17:24:38.910737 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903729    2577 flags.go:64] FLAG: --eviction-soft=""
Sep 16 17:24:38.910737 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903733    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 17:24:38.910737 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903738    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 17:24:38.910737 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903744    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 17:24:38.910737 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903749    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 17:24:38.910737 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903759    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 17:24:38.910737 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903763    2577 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 17:24:38.910737 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903768    2577 flags.go:64] FLAG: --feature-gates=""
Sep 16 17:24:38.910737 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903774    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 17:24:38.910737 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903779    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 17:24:38.910737 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903798    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 17:24:38.910737 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903805    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 17:24:38.910737 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903811    2577 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 17:24:38.910737 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903816    2577 flags.go:64] FLAG: --help="false"
Sep 16 17:24:38.910737 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903821    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-136-122.ec2.internal"
Sep 16 17:24:38.910737 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903827    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 17:24:38.910737 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903832    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 17:24:38.910737 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903837    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 17:24:38.911341 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903843    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 17:24:38.911341 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903848    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 17:24:38.911341 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903875    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 17:24:38.911341 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903881    2577 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 17:24:38.911341 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903886    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 17:24:38.911341 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903890    2577 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 17:24:38.911341 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903895    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 17:24:38.911341 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903900    2577 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 17:24:38.911341 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903905    2577 flags.go:64] FLAG: --kube-reserved=""
Sep 16 17:24:38.911341 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903910    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 17:24:38.911341 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903915    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 17:24:38.911341 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903920    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 17:24:38.911341 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903924    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 17:24:38.911341 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903929    2577 flags.go:64] FLAG: --lock-file=""
Sep 16 17:24:38.911341 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903934    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 17:24:38.911341 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903938    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 17:24:38.911341 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903943    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 17:24:38.911341 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903952    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 17:24:38.911341 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903957    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 17:24:38.911341 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903964    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 17:24:38.911341 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903970    2577 flags.go:64] FLAG: --logging-format="text"
Sep 16 17:24:38.911341 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903974    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 17:24:38.911341 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903980    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 17:24:38.911341 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903984    2577 flags.go:64] FLAG: --manifest-url=""
Sep 16 17:24:38.912021 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903989    2577 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 17:24:38.912021 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.903996    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 17:24:38.912021 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904000    2577 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 17:24:38.912021 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904007    2577 flags.go:64] FLAG: --max-pods="110"
Sep 16 17:24:38.912021 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904012    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 17:24:38.912021 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904017    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 17:24:38.912021 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904022    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 17:24:38.912021 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904027    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 17:24:38.912021 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904031    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 17:24:38.912021 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904036    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 17:24:38.912021 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904042    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 17:24:38.912021 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904054    2577 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 17:24:38.912021 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904070    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 17:24:38.912021 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904076    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 17:24:38.912021 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904081    2577 flags.go:64] FLAG: --pod-cidr=""
Sep 16 17:24:38.912021 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904087    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 17:24:38.912021 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904097    2577 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 17:24:38.912021 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904102    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 17:24:38.912021 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904107    2577 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 17:24:38.912021 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904111    2577 flags.go:64] FLAG: --port="10250"
Sep 16 17:24:38.912021 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904116    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 17:24:38.912021 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904121    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-040244dc907e01937"
Sep 16 17:24:38.912021 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904126    2577 flags.go:64] FLAG: --qos-reserved=""
Sep 16 17:24:38.912021 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904132    2577 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 17:24:38.912646 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904136    2577 flags.go:64] FLAG: --register-node="true"
Sep 16 17:24:38.912646 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904141    2577 flags.go:64] FLAG: --register-with-taints=""
Sep 16 17:24:38.912646 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904147    2577 flags.go:64] FLAG: --registry-burst="10"
Sep 16 17:24:38.912646 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904153    2577 flags.go:64] FLAG: --registry-qps="5"
Sep 16 17:24:38.912646 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904159    2577 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 17:24:38.912646 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904164    2577 flags.go:64] FLAG: --reserved-memory=""
Sep 16 17:24:38.912646 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904170    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 17:24:38.912646 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904175    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 17:24:38.912646 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904180    2577 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 17:24:38.912646 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904185    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 17:24:38.912646 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904190    2577 flags.go:64] FLAG: --runonce="false"
Sep 16 17:24:38.912646 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904194    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 17:24:38.912646 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904199    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 17:24:38.912646 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904204    2577 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 17:24:38.912646 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904210    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 17:24:38.912646 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904214    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 17:24:38.912646 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904230    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 17:24:38.912646 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904235    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 17:24:38.912646 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904242    2577 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 17:24:38.912646 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904246    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 17:24:38.912646 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904251    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 17:24:38.912646 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904256    2577 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 17:24:38.912646 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904261    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 17:24:38.912646 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904266    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 17:24:38.912646 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904271    2577 flags.go:64] FLAG: --system-cgroups=""
Sep 16 17:24:38.913291 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904275    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 17:24:38.913291 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904284    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 17:24:38.913291 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904289    2577 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 17:24:38.913291 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904294    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 17:24:38.913291 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904300    2577 flags.go:64] FLAG: --tls-min-version=""
Sep 16 17:24:38.913291 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904304    2577 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 17:24:38.913291 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904309    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 17:24:38.913291 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904314    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 17:24:38.913291 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904318    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 17:24:38.913291 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904323    2577 flags.go:64] FLAG: --v="2"
Sep 16 17:24:38.913291 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904329    2577 flags.go:64] FLAG: --version="false"
Sep 16 17:24:38.913291 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904338    2577 flags.go:64] FLAG: --vmodule=""
Sep 16 17:24:38.913291 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904344    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 17:24:38.913291 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.904350    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 17:24:38.913291 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904539    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 17:24:38.913291 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904546    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 17:24:38.913291 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904551    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 17:24:38.913291 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904556    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 17:24:38.913291 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904560    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 17:24:38.913291 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904564    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 17:24:38.913291 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904568    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 17:24:38.913291 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904573    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 17:24:38.913291 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904577    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 17:24:38.913856 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904584    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 17:24:38.913856 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904589    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 17:24:38.913856 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904596    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 17:24:38.913856 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904601    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 17:24:38.913856 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904605    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 17:24:38.913856 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904610    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 17:24:38.913856 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904614    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 17:24:38.913856 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904619    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 17:24:38.913856 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904624    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 17:24:38.913856 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904628    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 17:24:38.913856 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904633    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 17:24:38.913856 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904639    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 17:24:38.913856 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904643    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 17:24:38.913856 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904648    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 17:24:38.913856 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904652    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 17:24:38.913856 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904657    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 17:24:38.913856 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904661    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 17:24:38.913856 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904665    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 17:24:38.913856 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904669    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 17:24:38.914353 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904673    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 17:24:38.914353 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904680    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 17:24:38.914353 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904693    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 17:24:38.914353 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904698    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 17:24:38.914353 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904702    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 17:24:38.914353 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904706    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 17:24:38.914353 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904713    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 17:24:38.914353 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904719    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 17:24:38.914353 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904724    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 17:24:38.914353 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904728    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 17:24:38.914353 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904732    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 17:24:38.914353 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904736    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 17:24:38.914353 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904741    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 17:24:38.914353 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904745    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 17:24:38.914353 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904749    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 17:24:38.914353 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904755    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 17:24:38.914353 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904760    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 17:24:38.914353 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904764    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 17:24:38.914353 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904768    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 17:24:38.914353 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904772    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 17:24:38.914827 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904777    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 17:24:38.914827 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904781    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 17:24:38.914827 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904785    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 17:24:38.914827 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904789    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 17:24:38.914827 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904793    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 17:24:38.914827 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904797    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 17:24:38.914827 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904801    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 17:24:38.914827 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904806    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 17:24:38.914827 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904810    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 17:24:38.914827 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904814    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 17:24:38.914827 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904818    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 17:24:38.914827 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904823    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 17:24:38.914827 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904827    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 17:24:38.914827 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904832    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 17:24:38.914827 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904836    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 17:24:38.914827 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904842    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 17:24:38.914827 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904847    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 17:24:38.914827 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904854    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 17:24:38.914827 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904874    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 17:24:38.915310 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904879    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 17:24:38.915310 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904884    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 17:24:38.915310 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904890    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 17:24:38.915310 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904894    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 17:24:38.915310 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904898    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 17:24:38.915310 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904902    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 17:24:38.915310 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904906    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 17:24:38.915310 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904910    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 17:24:38.915310 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904917    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 17:24:38.915310 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904921    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 17:24:38.915310 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904925    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 17:24:38.915310 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904929    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 17:24:38.915310 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904934    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 17:24:38.915310 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904938    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 17:24:38.915310 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904942    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 17:24:38.915310 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904947    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 17:24:38.915310 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904951    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 17:24:38.915310 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904956    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 17:24:38.915310 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904960    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 17:24:38.915790 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904964    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 17:24:38.915790 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904969    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 17:24:38.915790 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904973    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 17:24:38.915790 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904977    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 17:24:38.915790 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904981    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 17:24:38.915790 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904986    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 17:24:38.915790 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904990    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 17:24:38.915790 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.904996    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 17:24:38.915790 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.905001    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 17:24:38.915790 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.905005    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 17:24:38.915790 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.905019    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 17:24:38.915790 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.905023    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 17:24:38.915790 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.905027    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 17:24:38.915790 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.905675    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 17:24:38.915790 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.913219    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 17:24:38.916223 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.913235    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 17:24:38.916223 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.913971    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 17:24:38.916223 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.913979    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 17:24:38.916223 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914033    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 17:24:38.916223 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914038    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 17:24:38.916223 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914041    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 17:24:38.916223 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914045    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 17:24:38.916223 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914048    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 17:24:38.916223 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914051    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 17:24:38.916223 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914055    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 17:24:38.916223 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914058    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 17:24:38.916223 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914060    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 17:24:38.916223 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914063    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 17:24:38.916223 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914066    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 17:24:38.916223 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914069    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 17:24:38.916223 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914072    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 17:24:38.916223 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914074    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 17:24:38.916223 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914077    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 17:24:38.916223 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914080    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 17:24:38.916693 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914082    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 17:24:38.916693 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914085    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 17:24:38.916693 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914089    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 17:24:38.916693 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914093    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 17:24:38.916693 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914096    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 17:24:38.916693 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914099    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 17:24:38.916693 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914103    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 17:24:38.916693 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914105    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 17:24:38.916693 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914110    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 17:24:38.916693 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914113    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 17:24:38.916693 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914116    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 17:24:38.916693 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914119    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 17:24:38.916693 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914122    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 17:24:38.916693 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914125    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 17:24:38.916693 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914129    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 17:24:38.916693 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914133    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 17:24:38.916693 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914136    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 17:24:38.916693 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914139    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 17:24:38.916693 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914142    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 17:24:38.917177 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914144    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 17:24:38.917177 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914147    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 17:24:38.917177 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914149    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 17:24:38.917177 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914152    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 17:24:38.917177 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914154    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 17:24:38.917177 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914157    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 17:24:38.917177 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914160    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 17:24:38.917177 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914162    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 17:24:38.917177 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914165    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 17:24:38.917177 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914168    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 17:24:38.917177 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914170    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 17:24:38.917177 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914173    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 17:24:38.917177 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914175    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 17:24:38.917177 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914178    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 17:24:38.917177 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914182    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 17:24:38.917177 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914185    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 17:24:38.917177 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914187    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 17:24:38.917177 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914190    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 17:24:38.917177 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914192    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 17:24:38.917731 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914195    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 17:24:38.917731 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914197    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 17:24:38.917731 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914200    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 17:24:38.917731 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914203    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 17:24:38.917731 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914206    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 17:24:38.917731 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914208    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 17:24:38.917731 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914211    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 17:24:38.917731 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914214    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 17:24:38.917731 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914216    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 17:24:38.917731 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914219    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 17:24:38.917731 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914222    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 17:24:38.917731 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914224    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 17:24:38.917731 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914226    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 17:24:38.917731 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914229    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 17:24:38.917731 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914232    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 17:24:38.917731 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914234    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 17:24:38.917731 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914238    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 17:24:38.917731 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914242    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 17:24:38.917731 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914245    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 17:24:38.918244 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914248    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 17:24:38.918244 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914251    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 17:24:38.918244 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914255    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 17:24:38.918244 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914257    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 17:24:38.918244 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914260    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 17:24:38.918244 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914263    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 17:24:38.918244 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914266    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 17:24:38.918244 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914268    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 17:24:38.918244 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914271    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 17:24:38.918244 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914274    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 17:24:38.918244 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914277    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 17:24:38.918244 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914280    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 17:24:38.918244 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914283    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 17:24:38.918244 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914285    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 17:24:38.918244 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914288    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 17:24:38.918244 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914290    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 17:24:38.918244 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914293    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 17:24:38.918244 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914296    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 17:24:38.918244 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914299    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 17:24:38.918244 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914302    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 17:24:38.918740 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914304    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 17:24:38.918740 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914307    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 17:24:38.918740 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914309    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 17:24:38.918740 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914312    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 17:24:38.918740 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914314    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 17:24:38.918740 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914317    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 17:24:38.918740 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.914321    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 17:24:38.918740 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914432    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 17:24:38.918740 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914436    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 17:24:38.918740 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914439    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 17:24:38.918740 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914442    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 17:24:38.918740 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914445    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 17:24:38.918740 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914448    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 17:24:38.918740 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914451    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 17:24:38.918740 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914454    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 17:24:38.919135 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914456    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 17:24:38.919135 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914459    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 17:24:38.919135 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914462    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 17:24:38.919135 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914464    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 17:24:38.919135 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914467    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 17:24:38.919135 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914469    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 17:24:38.919135 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914472    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 17:24:38.919135 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914475    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 17:24:38.919135 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914477    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 17:24:38.919135 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914480    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 17:24:38.919135 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914483    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 17:24:38.919135 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914485    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 17:24:38.919135 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914488    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 17:24:38.919135 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914491    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 17:24:38.919135 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914493    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 17:24:38.919135 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914496    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 17:24:38.919135 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914498    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 17:24:38.919135 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914501    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 17:24:38.919135 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914504    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 17:24:38.919135 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914506    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 17:24:38.919625 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914509    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 17:24:38.919625 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914511    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 17:24:38.919625 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914514    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 17:24:38.919625 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914516    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 17:24:38.919625 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914519    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 17:24:38.919625 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914521    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 17:24:38.919625 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914524    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 17:24:38.919625 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914526    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 17:24:38.919625 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914528    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 17:24:38.919625 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914531    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 17:24:38.919625 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914533    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 17:24:38.919625 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914536    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 17:24:38.919625 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914538    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 17:24:38.919625 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914541    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 17:24:38.919625 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914543    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 17:24:38.919625 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914546    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 17:24:38.919625 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914549    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 17:24:38.919625 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914551    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 17:24:38.919625 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914554    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 17:24:38.919625 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914557    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 17:24:38.920131 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914559    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 17:24:38.920131 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914562    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 17:24:38.920131 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914564    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 17:24:38.920131 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914567    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 17:24:38.920131 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914569    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 17:24:38.920131 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914572    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 17:24:38.920131 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914574    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 17:24:38.920131 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914577    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 17:24:38.920131 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914580    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 17:24:38.920131 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914583    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 17:24:38.920131 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914586    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 17:24:38.920131 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914588    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 17:24:38.920131 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914592    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 17:24:38.920131 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914596    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 17:24:38.920131 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914598    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 17:24:38.920131 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914601    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 17:24:38.920131 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914605    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 17:24:38.920131 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914608    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 17:24:38.920131 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914611    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 17:24:38.920595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914614    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 17:24:38.920595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914617    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 17:24:38.920595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914620    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 17:24:38.920595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914622    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 17:24:38.920595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914625    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 17:24:38.920595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914627    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 17:24:38.920595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914630    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 17:24:38.920595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914632    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 17:24:38.920595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914635    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 17:24:38.920595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914637    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 17:24:38.920595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914640    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 17:24:38.920595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914643    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 17:24:38.920595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914645    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 17:24:38.920595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914648    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 17:24:38.920595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914650    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 17:24:38.920595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914653    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 17:24:38.920595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914656    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 17:24:38.920595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914658    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 17:24:38.920595 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914661    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 17:24:38.921076 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914663    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 17:24:38.921076 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914666    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 17:24:38.921076 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914669    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 17:24:38.921076 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914672    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 17:24:38.921076 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914675    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 17:24:38.921076 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914677    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 17:24:38.921076 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914680    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 17:24:38.921076 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914682    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 17:24:38.921076 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914685    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 17:24:38.921076 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914689    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 17:24:38.921076 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914692    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 17:24:38.921076 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914695    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 17:24:38.921076 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:38.914697    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 17:24:38.921076 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.914702    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 17:24:38.921076 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.914799    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 17:24:38.921447 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.917005    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 17:24:38.921447 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.918004    2577 server.go:1019] "Starting client certificate rotation"
Sep 16 17:24:38.921447 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.918104    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 17:24:38.921447 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.918141    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 17:24:38.921447 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.920439    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 17:24:38.924550 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.924534    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 17:24:38.933158 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.933139    2577 log.go:25] "Validated CRI v1 runtime API"
Sep 16 17:24:38.937030 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.936169    2577 log.go:25] "Validated CRI v1 image API"
Sep 16 17:24:38.937834 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.937817    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 17:24:38.942349 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.942331    2577 fs.go:135] Filesystem UUIDs: map[132c50a4-b37d-4408-99df-4de05690730a:/dev/nvme0n1p3 626bb702-5095-4df1-be98-e1f2f8294ee6:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 16 17:24:38.942419 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.942348    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 17:24:38.947397 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.947277    2577 manager.go:216] Machine: {Timestamp:2026-09-16 17:24:38.945992185 +0000 UTC m=+0.308670962 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3145015 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27594fd64032cba02ed550ab42ccc0 SystemUUID:ec27594f-d640-32cb-a02e-d550ab42ccc0 BootID:7eadcae2-76ed-49e6-97fc-dc6feacaf99a Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d1:45:db:03 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d1:45:db:03 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:b7:05:0e:dc:47 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 17:24:38.947397 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.947390    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 17:24:38.947521 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.947471    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 17:24:38.950499 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.950473    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 17:24:38.950624 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.950501    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-122.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 17:24:38.951723 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.951712    2577 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 17:24:38.951771 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.951726    2577 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 17:24:38.951771 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.951734    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 17:24:38.952419 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.952410    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 17:24:38.953349 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.953338    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 17:24:38.953643 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.953628    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 17:24:38.957550 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.957538    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 17:24:38.957658 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.957649    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 17:24:38.957689 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.957679    2577 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 17:24:38.957717 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.957692    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 17:24:38.957744 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.957716    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 17:24:38.957744 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.957731    2577 kubelet.go:398] "Adding apiserver pod source"
Sep 16 17:24:38.957744 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.957742    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 17:24:38.958267 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.958254    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 17:24:38.958311 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.958274    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 17:24:38.960228 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.960209    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 17:24:38.962410 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.962393    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 17:24:38.962462 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.962429    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 17:24:38.964106 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.964091    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 17:24:38.964152 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.964109    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 17:24:38.964152 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.964115    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 17:24:38.964152 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.964121    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 17:24:38.964152 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.964129    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 17:24:38.964152 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.964135    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 17:24:38.964152 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.964152    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 17:24:38.964311 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.964157    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 17:24:38.964311 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.964164    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 17:24:38.964311 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.964170    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 17:24:38.964311 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.964178    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 17:24:38.964311 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.964183    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 17:24:38.964929 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.964917    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 17:24:38.964992 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.964935    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 17:24:38.968310 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.968289    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-122.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 17:24:38.968389 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:38.968361    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-122.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 17:24:38.968389 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:38.968361    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 17:24:38.969001 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.968987    2577 server.go:1268] "Started kubelet"
Sep 16 17:24:38.969101 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.969072    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 17:24:38.969260 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.969215    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 17:24:38.969325 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.969317    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 17:24:38.969981 ip-10-0-136-122 systemd[1]: Started Kubernetes Kubelet.
Sep 16 17:24:38.971275 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.971256    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 17:24:38.974449 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.974435    2577 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 17:24:38.974649 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:38.974608    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 17:24:38.975440 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:38.974602    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-122.ec2.internal.18d5dd0df54604ad  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-122.ec2.internal,UID:ip-10-0-136-122.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-122.ec2.internal,},FirstTimestamp:2026-09-16 17:24:38.968960173 +0000 UTC m=+0.331638950,LastTimestamp:2026-09-16 17:24:38.968960173 +0000 UTC m=+0.331638950,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-122.ec2.internal,}"
Sep 16 17:24:38.977992 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.977963    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 17:24:38.978483 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.978466    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 17:24:38.978589 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:38.978566    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-122.ec2.internal\" not found"
Sep 16 17:24:38.979550 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.979343    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 17:24:38.979550 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.979355    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 17:24:38.979660 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.979563    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 17:24:38.980577 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.980558    2577 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 17:24:38.980577 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.980575    2577 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 17:24:38.980832 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.980816    2577 factory.go:55] Registering systemd factory
Sep 16 17:24:38.980927 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.980835    2577 factory.go:223] Registration of the systemd container factory successfully
Sep 16 17:24:38.981793 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.981777    2577 factory.go:153] Registering CRI-O factory
Sep 16 17:24:38.981793 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.981794    2577 factory.go:223] Registration of the crio container factory successfully
Sep 16 17:24:38.981952 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.981891    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 17:24:38.982068 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.982052    2577 factory.go:103] Registering Raw factory
Sep 16 17:24:38.982175 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.982086    2577 manager.go:1196] Started watching for new ooms in manager
Sep 16 17:24:38.982971 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.982951    2577 manager.go:349] Starting recovery of all containers
Sep 16 17:24:38.987223 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:38.987193    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 17:24:38.987652 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:38.987545    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-122.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 17:24:38.990437 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.990410    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lw9m8"
Sep 16 17:24:38.993858 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.993843    2577 manager.go:354] Recovery completed
Sep 16 17:24:38.995033 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:38.995010    2577 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 16 17:24:38.998485 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.998471    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:38.999337 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:38.999321    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lw9m8"
Sep 16 17:24:39.000173 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.000159    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:39.000247 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.000186    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:39.000247 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.000196    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-122.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:39.000844 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.000828    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 17:24:39.000844 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.000843    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 17:24:39.000977 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.000833    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:39.000977 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.000876    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 17:24:39.001610 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.001593    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:39.001687 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.001637    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:39.001687 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.001663    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-122.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:39.002678 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:39.002582    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-122.ec2.internal.18d5dd0df7224a11  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-122.ec2.internal,UID:ip-10-0-136-122.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-122.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-122.ec2.internal,},FirstTimestamp:2026-09-16 17:24:39.000173073 +0000 UTC m=+0.362851870,LastTimestamp:2026-09-16 17:24:39.000173073 +0000 UTC m=+0.362851870,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-122.ec2.internal,}"
Sep 16 17:24:39.003066 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.003053    2577 policy_none.go:49] "None policy: Start"
Sep 16 17:24:39.003126 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.003072    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 17:24:39.003126 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.003084    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 17:24:39.005269 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.005104    2577 policy_none.go:47] "Start"
Sep 16 17:24:39.054016 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.045437    2577 manager.go:337] "Starting Device Plugin manager"
Sep 16 17:24:39.054016 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:39.045476    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 17:24:39.054016 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.045486    2577 server.go:87] "Starting device plugin registration server"
Sep 16 17:24:39.054016 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.045767    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 17:24:39.054016 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.045779    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 17:24:39.054016 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.045896    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 17:24:39.054016 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.045992    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 17:24:39.054016 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.046003    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 17:24:39.054016 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:39.046575    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 17:24:39.054016 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:39.046626    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-122.ec2.internal\" not found"
Sep 16 17:24:39.072706 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.072671    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 17:24:39.073841 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.073821    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 17:24:39.073941 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.073885    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 17:24:39.073941 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.073916    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 17:24:39.074031 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:39.074007    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 17:24:39.076170 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.076150    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 17:24:39.145960 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.145848    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:39.147523 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.147507    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:39.147615 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.147534    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:39.147615 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.147546    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-122.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:39.147615 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.147579    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-122.ec2.internal"
Sep 16 17:24:39.160926 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.160901    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-122.ec2.internal"
Sep 16 17:24:39.161039 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:39.160936    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-122.ec2.internal\": node \"ip-10-0-136-122.ec2.internal\" not found"
Sep 16 17:24:39.174652 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.174056    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-122.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-122.ec2.internal"]
Sep 16 17:24:39.174773 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.174689    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:39.175499 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.175483    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:39.175576 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.175515    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:39.175576 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.175530    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-122.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:39.175666 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:39.175628    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-122.ec2.internal\" not found"
Sep 16 17:24:39.177928 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.177916    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:39.178058 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.178036    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-122.ec2.internal"
Sep 16 17:24:39.178096 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.178071    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:39.178601 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.178586    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:39.178653 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.178612    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:39.178653 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.178626    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-122.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:39.178653 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.178637    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:39.178748 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.178654    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:39.178748 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.178666    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-122.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:39.180956 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.180940    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-122.ec2.internal"
Sep 16 17:24:39.181037 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.180965    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:39.181037 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.181007    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/80b06e6b7e68304bfaedbddff98b161c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-122.ec2.internal\" (UID: \"80b06e6b7e68304bfaedbddff98b161c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-122.ec2.internal"
Sep 16 17:24:39.181117 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.181037    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/80b06e6b7e68304bfaedbddff98b161c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-122.ec2.internal\" (UID: \"80b06e6b7e68304bfaedbddff98b161c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-122.ec2.internal"
Sep 16 17:24:39.181117 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.181057    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b8e15c478aefab14292c7bbfa4653ed5-config\") pod \"kube-apiserver-proxy-ip-10-0-136-122.ec2.internal\" (UID: \"b8e15c478aefab14292c7bbfa4653ed5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-122.ec2.internal"
Sep 16 17:24:39.181612 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.181594    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:39.181702 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.181618    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:39.181702 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.181630    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-122.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:39.204546 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:39.204529    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-122.ec2.internal\" not found" node="ip-10-0-136-122.ec2.internal"
Sep 16 17:24:39.208838 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:39.208820    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-122.ec2.internal\" not found" node="ip-10-0-136-122.ec2.internal"
Sep 16 17:24:39.276168 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:39.276140    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-122.ec2.internal\" not found"
Sep 16 17:24:39.281481 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.281466    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/80b06e6b7e68304bfaedbddff98b161c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-122.ec2.internal\" (UID: \"80b06e6b7e68304bfaedbddff98b161c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-122.ec2.internal"
Sep 16 17:24:39.281557 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.281488    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/80b06e6b7e68304bfaedbddff98b161c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-122.ec2.internal\" (UID: \"80b06e6b7e68304bfaedbddff98b161c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-122.ec2.internal"
Sep 16 17:24:39.281557 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.281505    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b8e15c478aefab14292c7bbfa4653ed5-config\") pod \"kube-apiserver-proxy-ip-10-0-136-122.ec2.internal\" (UID: \"b8e15c478aefab14292c7bbfa4653ed5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-122.ec2.internal"
Sep 16 17:24:39.281622 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.281571    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b8e15c478aefab14292c7bbfa4653ed5-config\") pod \"kube-apiserver-proxy-ip-10-0-136-122.ec2.internal\" (UID: \"b8e15c478aefab14292c7bbfa4653ed5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-122.ec2.internal"
Sep 16 17:24:39.281622 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.281576    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/80b06e6b7e68304bfaedbddff98b161c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-122.ec2.internal\" (UID: \"80b06e6b7e68304bfaedbddff98b161c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-122.ec2.internal"
Sep 16 17:24:39.281622 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.281576    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/80b06e6b7e68304bfaedbddff98b161c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-122.ec2.internal\" (UID: \"80b06e6b7e68304bfaedbddff98b161c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-122.ec2.internal"
Sep 16 17:24:39.376732 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:39.376680    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-122.ec2.internal\" not found"
Sep 16 17:24:39.477582 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:39.477510    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-122.ec2.internal\" not found"
Sep 16 17:24:39.508019 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.507991    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-122.ec2.internal"
Sep 16 17:24:39.511918 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.511901    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-122.ec2.internal"
Sep 16 17:24:39.578218 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:39.578175    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-122.ec2.internal\" not found"
Sep 16 17:24:39.679178 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:39.679129    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-122.ec2.internal\" not found"
Sep 16 17:24:39.782556 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:39.782456    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-122.ec2.internal\" not found"
Sep 16 17:24:39.884845 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:39.884795    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-122.ec2.internal\" not found"
Sep 16 17:24:39.918391 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.918359    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 17:24:39.918930 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:39.918504    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 17:24:39.984957 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:39.984928    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-122.ec2.internal\" not found"
Sep 16 17:24:40.005026 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.004987    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 17:19:39 +0000 UTC" deadline="2028-05-30 02:49:00.56987012 +0000 UTC"
Sep 16 17:24:40.005026 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.005017    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14913h24m20.564856061s"
Sep 16 17:24:40.085262 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:40.085194    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-122.ec2.internal\" not found"
Sep 16 17:24:40.141978 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:40.141943    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b06e6b7e68304bfaedbddff98b161c.slice/crio-f69fe7718648346f7aa9548fc2b8697394fd7df9d740909880e14fc28b90750e WatchSource:0}: Error finding container f69fe7718648346f7aa9548fc2b8697394fd7df9d740909880e14fc28b90750e: Status 404 returned error can't find the container with id f69fe7718648346f7aa9548fc2b8697394fd7df9d740909880e14fc28b90750e
Sep 16 17:24:40.147681 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.147662    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 17:24:40.153163 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:40.153141    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8e15c478aefab14292c7bbfa4653ed5.slice/crio-756e9deb1104627c851b4c6a60acadcf9dc781171097db9aa65854140f532bca WatchSource:0}: Error finding container 756e9deb1104627c851b4c6a60acadcf9dc781171097db9aa65854140f532bca: Status 404 returned error can't find the container with id 756e9deb1104627c851b4c6a60acadcf9dc781171097db9aa65854140f532bca
Sep 16 17:24:40.156787 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.156770    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 17:24:40.185950 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:40.185916    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-122.ec2.internal\" not found"
Sep 16 17:24:40.199308 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.199286    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 17:24:40.284951 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.284916    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-122.ec2.internal"
Sep 16 17:24:40.298583 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.298560    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 17:24:40.299577 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.299562    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-122.ec2.internal"
Sep 16 17:24:40.310382 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.310365    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 17:24:40.440572 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.440492    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 17:24:40.959077 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.959042    2577 apiserver.go:52] "Watching apiserver"
Sep 16 17:24:40.968445 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.968415    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 17:24:40.968949 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.968921    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-dwmgh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-122.ec2.internal","openshift-multus/multus-additional-cni-plugins-njrdx","openshift-multus/network-metrics-daemon-shvlb","openshift-network-diagnostics/network-check-target-wklqk","openshift-network-operator/iptables-alerter-j2xzd","kube-system/konnectivity-agent-9k478","kube-system/kube-apiserver-proxy-ip-10-0-136-122.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms5pm","openshift-cluster-node-tuning-operator/tuned-jdgc7","openshift-multus/multus-7rnft","openshift-ovn-kubernetes/ovnkube-node-5btzs","kube-system/global-pull-secret-syncer-6dj5q","openshift-dns/node-resolver-jfsjw"]
Sep 16 17:24:40.972030 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.971631    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9k478"
Sep 16 17:24:40.974243 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.974194    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7rnft"
Sep 16 17:24:40.974982 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.974956    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mwpng\""
Sep 16 17:24:40.975562 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.975374    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 17:24:40.975562 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.975501    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 17:24:40.976568 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.976547    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-n7cgl\""
Sep 16 17:24:40.976989 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.976749    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 17:24:40.977168 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.977144    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-njrdx"
Sep 16 17:24:40.977378 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.977321    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 17:24:40.977551 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.977518    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 17:24:40.977827 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.977799    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 17:24:40.979090 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.978800    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 17:24:40.981419 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.979584    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 17:24:40.981419 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.980534    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:24:40.981419 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:40.980953    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shvlb" podUID="0e392d83-bd3a-4d0b-ac4f-300a6019c434"
Sep 16 17:24:40.981419 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.981097    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 17:24:40.981419 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.981118    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8ksm4\""
Sep 16 17:24:40.984267 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.984240    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wklqk"
Sep 16 17:24:40.984368 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:40.984330    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wklqk" podUID="5d5f021d-6241-485a-8611-9fd0beb2ac98"
Sep 16 17:24:40.986485 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.986463    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j2xzd"
Sep 16 17:24:40.988107 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.988082    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74d1b87b-73ed-4d75-b943-e2f8a13808be-cnibin\") pod \"multus-additional-cni-plugins-njrdx\" (UID: \"74d1b87b-73ed-4d75-b943-e2f8a13808be\") " pod="openshift-multus/multus-additional-cni-plugins-njrdx"
Sep 16 17:24:40.988202 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.988122    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74d1b87b-73ed-4d75-b943-e2f8a13808be-tuning-conf-dir\") pod \"multus-additional-cni-plugins-njrdx\" (UID: \"74d1b87b-73ed-4d75-b943-e2f8a13808be\") " pod="openshift-multus/multus-additional-cni-plugins-njrdx"
Sep 16 17:24:40.988202 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.988151    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj2t8\" (UniqueName: \"kubernetes.io/projected/74d1b87b-73ed-4d75-b943-e2f8a13808be-kube-api-access-cj2t8\") pod \"multus-additional-cni-plugins-njrdx\" (UID: \"74d1b87b-73ed-4d75-b943-e2f8a13808be\") " pod="openshift-multus/multus-additional-cni-plugins-njrdx"
Sep 16 17:24:40.988202 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.988178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxv2t\" (UniqueName: \"kubernetes.io/projected/0e392d83-bd3a-4d0b-ac4f-300a6019c434-kube-api-access-rxv2t\") pod \"network-metrics-daemon-shvlb\" (UID: \"0e392d83-bd3a-4d0b-ac4f-300a6019c434\") " pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:24:40.988385 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.988203    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c910d08c-80cd-486a-98b3-518d769d149f-konnectivity-ca\") pod \"konnectivity-agent-9k478\" (UID: \"c910d08c-80cd-486a-98b3-518d769d149f\") " pod="kube-system/konnectivity-agent-9k478"
Sep 16 17:24:40.988385 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.988228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-system-cni-dir\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:40.988385 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.988252    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-host-run-k8s-cni-cncf-io\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:40.988385 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.988282    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-host-var-lib-cni-bin\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:40.988385 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.988309    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd4c6\" (UniqueName: \"kubernetes.io/projected/5d5f021d-6241-485a-8611-9fd0beb2ac98-kube-api-access-jd4c6\") pod \"network-check-target-wklqk\" (UID: \"5d5f021d-6241-485a-8611-9fd0beb2ac98\") " pod="openshift-network-diagnostics/network-check-target-wklqk"
Sep 16 17:24:40.988385 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.988353    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-host-run-multus-certs\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:40.988674 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.988401    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74d1b87b-73ed-4d75-b943-e2f8a13808be-system-cni-dir\") pod \"multus-additional-cni-plugins-njrdx\" (UID: \"74d1b87b-73ed-4d75-b943-e2f8a13808be\") " pod="openshift-multus/multus-additional-cni-plugins-njrdx"
Sep 16 17:24:40.988674 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.988432    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b674965e-c124-4b4c-ae07-0e17f355f0e0-cni-binary-copy\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:40.988674 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.988462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-hostroot\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:40.988674 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.988489    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b674965e-c124-4b4c-ae07-0e17f355f0e0-multus-daemon-config\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:40.988674 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.988548    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-host-var-lib-kubelet\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:40.988919 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.988693    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/74d1b87b-73ed-4d75-b943-e2f8a13808be-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-njrdx\" (UID: \"74d1b87b-73ed-4d75-b943-e2f8a13808be\") " pod="openshift-multus/multus-additional-cni-plugins-njrdx"
Sep 16 17:24:40.988919 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.988726    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-multus-cni-dir\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:40.988919 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.988751    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-etc-kubernetes\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:40.988919 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.988775    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-multus-conf-dir\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:40.988919 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.988800    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmlq7\" (UniqueName: \"kubernetes.io/projected/b674965e-c124-4b4c-ae07-0e17f355f0e0-kube-api-access-jmlq7\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:40.988919 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.988828    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/74d1b87b-73ed-4d75-b943-e2f8a13808be-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-njrdx\" (UID: \"74d1b87b-73ed-4d75-b943-e2f8a13808be\") " pod="openshift-multus/multus-additional-cni-plugins-njrdx"
Sep 16 17:24:40.988919 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.988855    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e392d83-bd3a-4d0b-ac4f-300a6019c434-metrics-certs\") pod \"network-metrics-daemon-shvlb\" (UID: \"0e392d83-bd3a-4d0b-ac4f-300a6019c434\") " pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:24:40.988919 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.988878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c910d08c-80cd-486a-98b3-518d769d149f-agent-certs\") pod \"konnectivity-agent-9k478\" (UID: \"c910d08c-80cd-486a-98b3-518d769d149f\") " pod="kube-system/konnectivity-agent-9k478"
Sep 16 17:24:40.988919 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.988923    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-multus-socket-dir-parent\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:40.989387 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.988949    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-host-var-lib-cni-multus\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:40.989387 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.988974    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74d1b87b-73ed-4d75-b943-e2f8a13808be-cni-binary-copy\") pod \"multus-additional-cni-plugins-njrdx\" (UID: \"74d1b87b-73ed-4d75-b943-e2f8a13808be\") " pod="openshift-multus/multus-additional-cni-plugins-njrdx"
Sep 16 17:24:40.989387 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.988998    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-cnibin\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:40.989387 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.989030    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-host-run-netns\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:40.989387 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.989207    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 17:24:40.989387 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.989276    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms5pm"
Sep 16 17:24:40.989917 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.989896    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 17:24:40.990016 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.989922    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wwr9h\""
Sep 16 17:24:40.990196 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.990178    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 17:24:40.991545 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.991524    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jfsjw"
Sep 16 17:24:40.993239 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.992866    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 17:24:40.993239 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.992883    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 17:24:40.993239 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.993119    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-749hx\""
Sep 16 17:24:40.993239 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.993232    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 17:24:40.993910 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.993890    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 17:24:40.994414 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.994394    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lnjxd\""
Sep 16 17:24:40.994504 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.994473    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 17:24:40.994709 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.994691    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:40.996122 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.996025    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 17:24:40.997143 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.997122    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 17:24:40.997387 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.997369    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 17:24:40.997651 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.997634    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:40.998597 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.998573    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bvgbv\""
Sep 16 17:24:40.998983 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.998713    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 17:24:40.998983 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.998784    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 17:24:40.998983 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.998791    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 17:24:40.998983 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:40.998785    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 17:24:41.000285 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.000036    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jhh58\""
Sep 16 17:24:41.000285 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.000213    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 17:24:41.000457 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.000294    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 17:24:41.000536 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.000513    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dwmgh"
Sep 16 17:24:41.003130 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.003110    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:24:41.003225 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:41.003183    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dj5q" podUID="451de579-98aa-48be-aeee-cf7d09759d3e"
Sep 16 17:24:41.004353 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.004333    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 17:24:41.004563 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.004546    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dz7gd\""
Sep 16 17:24:41.004998 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.004771    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 17:24:41.004998 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.004796    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 17:24:41.043553 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.043524    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-h7lmm"
Sep 16 17:24:41.053596 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.053571    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-h7lmm"
Sep 16 17:24:41.079787 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.079734    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-122.ec2.internal" event={"ID":"b8e15c478aefab14292c7bbfa4653ed5","Type":"ContainerStarted","Data":"756e9deb1104627c851b4c6a60acadcf9dc781171097db9aa65854140f532bca"}
Sep 16 17:24:41.080480 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.080456    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 17:24:41.081263 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.081224    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-122.ec2.internal" event={"ID":"80b06e6b7e68304bfaedbddff98b161c","Type":"ContainerStarted","Data":"f69fe7718648346f7aa9548fc2b8697394fd7df9d740909880e14fc28b90750e"}
Sep 16 17:24:41.089379 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.089353    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/445899b8-af25-4084-a9c8-d917110aea4b-iptables-alerter-script\") pod \"iptables-alerter-j2xzd\" (UID: \"445899b8-af25-4084-a9c8-d917110aea4b\") " pod="openshift-network-operator/iptables-alerter-j2xzd"
Sep 16 17:24:41.089494 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.089386    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/445899b8-af25-4084-a9c8-d917110aea4b-host-slash\") pod \"iptables-alerter-j2xzd\" (UID: \"445899b8-af25-4084-a9c8-d917110aea4b\") " pod="openshift-network-operator/iptables-alerter-j2xzd"
Sep 16 17:24:41.089494 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.089412    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-host-run-netns\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.089494 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.089434    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-host-run-ovn-kubernetes\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.089494 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.089466    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e392d83-bd3a-4d0b-ac4f-300a6019c434-metrics-certs\") pod \"network-metrics-daemon-shvlb\" (UID: \"0e392d83-bd3a-4d0b-ac4f-300a6019c434\") " pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:24:41.089689 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.089502    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xr22\" (UniqueName: \"kubernetes.io/projected/445899b8-af25-4084-a9c8-d917110aea4b-kube-api-access-5xr22\") pod \"iptables-alerter-j2xzd\" (UID: \"445899b8-af25-4084-a9c8-d917110aea4b\") " pod="openshift-network-operator/iptables-alerter-j2xzd"
Sep 16 17:24:41.089689 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.089537    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-var-lib-openvswitch\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.089689 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.089567    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-node-log\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.089689 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.089594    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a202ee1a-5714-4706-9464-7076448f3bf0-sys\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.089689 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:41.089606    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:41.089689 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.089622    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74d1b87b-73ed-4d75-b943-e2f8a13808be-cni-binary-copy\") pod \"multus-additional-cni-plugins-njrdx\" (UID: \"74d1b87b-73ed-4d75-b943-e2f8a13808be\") " pod="openshift-multus/multus-additional-cni-plugins-njrdx"
Sep 16 17:24:41.089689 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.089646    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-host-run-netns\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:41.089689 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:41.089687    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e392d83-bd3a-4d0b-ac4f-300a6019c434-metrics-certs podName:0e392d83-bd3a-4d0b-ac4f-300a6019c434 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:41.589661485 +0000 UTC m=+2.952340274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e392d83-bd3a-4d0b-ac4f-300a6019c434-metrics-certs") pod "network-metrics-daemon-shvlb" (UID: "0e392d83-bd3a-4d0b-ac4f-300a6019c434") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:41.090124 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.089707    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-host-run-netns\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:41.090124 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.089715    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a202ee1a-5714-4706-9464-7076448f3bf0-etc-sysctl-d\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.090124 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.089752    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwrpw\" (UniqueName: \"kubernetes.io/projected/4c1797a6-6455-460e-a992-8bcd6171acff-kube-api-access-lwrpw\") pod \"aws-ebs-csi-driver-node-ms5pm\" (UID: \"4c1797a6-6455-460e-a992-8bcd6171acff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms5pm"
Sep 16 17:24:41.090124 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.089779    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffd13091-dadb-4b9f-bd59-53f1073acedf-host\") pod \"node-ca-dwmgh\" (UID: \"ffd13091-dadb-4b9f-bd59-53f1073acedf\") " pod="openshift-image-registry/node-ca-dwmgh"
Sep 16 17:24:41.090124 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.089809    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74d1b87b-73ed-4d75-b943-e2f8a13808be-cnibin\") pod \"multus-additional-cni-plugins-njrdx\" (UID: \"74d1b87b-73ed-4d75-b943-e2f8a13808be\") " pod="openshift-multus/multus-additional-cni-plugins-njrdx"
Sep 16 17:24:41.090124 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.089839    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj2t8\" (UniqueName: \"kubernetes.io/projected/74d1b87b-73ed-4d75-b943-e2f8a13808be-kube-api-access-cj2t8\") pod \"multus-additional-cni-plugins-njrdx\" (UID: \"74d1b87b-73ed-4d75-b943-e2f8a13808be\") " pod="openshift-multus/multus-additional-cni-plugins-njrdx"
Sep 16 17:24:41.090124 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.089865    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-host-run-k8s-cni-cncf-io\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:41.090124 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.089893    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74d1b87b-73ed-4d75-b943-e2f8a13808be-cnibin\") pod \"multus-additional-cni-plugins-njrdx\" (UID: \"74d1b87b-73ed-4d75-b943-e2f8a13808be\") " pod="openshift-multus/multus-additional-cni-plugins-njrdx"
Sep 16 17:24:41.090124 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.089891    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-log-socket\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.090124 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.089943    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-ovn-node-metrics-cert\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.090124 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.089968    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4c1797a6-6455-460e-a992-8bcd6171acff-socket-dir\") pod \"aws-ebs-csi-driver-node-ms5pm\" (UID: \"4c1797a6-6455-460e-a992-8bcd6171acff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms5pm"
Sep 16 17:24:41.090124 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.089961    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-host-run-k8s-cni-cncf-io\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:41.090124 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.089996    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2r7g\" (UniqueName: \"kubernetes.io/projected/c6e2238e-ef6c-463e-8123-4663b5961709-kube-api-access-m2r7g\") pod \"node-resolver-jfsjw\" (UID: \"c6e2238e-ef6c-463e-8123-4663b5961709\") " pod="openshift-dns/node-resolver-jfsjw"
Sep 16 17:24:41.090124 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090025    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd4c6\" (UniqueName: \"kubernetes.io/projected/5d5f021d-6241-485a-8611-9fd0beb2ac98-kube-api-access-jd4c6\") pod \"network-check-target-wklqk\" (UID: \"5d5f021d-6241-485a-8611-9fd0beb2ac98\") " pod="openshift-network-diagnostics/network-check-target-wklqk"
Sep 16 17:24:41.090124 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090054    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-host-slash\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.090124 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090077    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/451de579-98aa-48be-aeee-cf7d09759d3e-dbus\") pod \"global-pull-secret-syncer-6dj5q\" (UID: \"451de579-98aa-48be-aeee-cf7d09759d3e\") " pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:24:41.090124 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090105    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74d1b87b-73ed-4d75-b943-e2f8a13808be-system-cni-dir\") pod \"multus-additional-cni-plugins-njrdx\" (UID: \"74d1b87b-73ed-4d75-b943-e2f8a13808be\") " pod="openshift-multus/multus-additional-cni-plugins-njrdx"
Sep 16 17:24:41.090871 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090129    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-etc-openvswitch\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.090871 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090152    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-run-openvswitch\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.090871 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090162    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74d1b87b-73ed-4d75-b943-e2f8a13808be-system-cni-dir\") pod \"multus-additional-cni-plugins-njrdx\" (UID: \"74d1b87b-73ed-4d75-b943-e2f8a13808be\") " pod="openshift-multus/multus-additional-cni-plugins-njrdx"
Sep 16 17:24:41.090871 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090177    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-run-ovn\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.090871 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090200    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a202ee1a-5714-4706-9464-7076448f3bf0-run\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.090871 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090226    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a202ee1a-5714-4706-9464-7076448f3bf0-lib-modules\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.090871 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090250    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a202ee1a-5714-4706-9464-7076448f3bf0-var-lib-kubelet\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.090871 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090294    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74d1b87b-73ed-4d75-b943-e2f8a13808be-cni-binary-copy\") pod \"multus-additional-cni-plugins-njrdx\" (UID: \"74d1b87b-73ed-4d75-b943-e2f8a13808be\") " pod="openshift-multus/multus-additional-cni-plugins-njrdx"
Sep 16 17:24:41.090871 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090301    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6e2238e-ef6c-463e-8123-4663b5961709-tmp-dir\") pod \"node-resolver-jfsjw\" (UID: \"c6e2238e-ef6c-463e-8123-4663b5961709\") " pod="openshift-dns/node-resolver-jfsjw"
Sep 16 17:24:41.090871 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090387    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-multus-cni-dir\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:41.090871 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-etc-kubernetes\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:41.090871 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090442    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-env-overrides\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.090871 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a202ee1a-5714-4706-9464-7076448f3bf0-tmp\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.090871 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090481    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4c1797a6-6455-460e-a992-8bcd6171acff-device-dir\") pod \"aws-ebs-csi-driver-node-ms5pm\" (UID: \"4c1797a6-6455-460e-a992-8bcd6171acff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms5pm"
Sep 16 17:24:41.090871 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090505    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c6e2238e-ef6c-463e-8123-4663b5961709-hosts-file\") pod \"node-resolver-jfsjw\" (UID: \"c6e2238e-ef6c-463e-8123-4663b5961709\") " pod="openshift-dns/node-resolver-jfsjw"
Sep 16 17:24:41.090871 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090515    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-etc-kubernetes\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:41.090871 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090532    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-multus-conf-dir\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:41.091717 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090548    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-multus-cni-dir\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:41.091717 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090558    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a202ee1a-5714-4706-9464-7076448f3bf0-etc-sysctl-conf\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.091717 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090585    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-multus-conf-dir\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:41.091717 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090622    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4c1797a6-6455-460e-a992-8bcd6171acff-etc-selinux\") pod \"aws-ebs-csi-driver-node-ms5pm\" (UID: \"4c1797a6-6455-460e-a992-8bcd6171acff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms5pm"
Sep 16 17:24:41.091717 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090662    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/74d1b87b-73ed-4d75-b943-e2f8a13808be-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-njrdx\" (UID: \"74d1b87b-73ed-4d75-b943-e2f8a13808be\") " pod="openshift-multus/multus-additional-cni-plugins-njrdx"
Sep 16 17:24:41.091717 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090708    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c910d08c-80cd-486a-98b3-518d769d149f-agent-certs\") pod \"konnectivity-agent-9k478\" (UID: \"c910d08c-80cd-486a-98b3-518d769d149f\") " pod="kube-system/konnectivity-agent-9k478"
Sep 16 17:24:41.091717 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-multus-socket-dir-parent\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:41.091717 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090768    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-host-var-lib-cni-multus\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:41.091717 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090829    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-ovnkube-config\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.091717 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090858    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-host-var-lib-cni-multus\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:41.091717 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090861    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-multus-socket-dir-parent\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:41.091717 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090913    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-cnibin\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:41.091717 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090940    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-systemd-units\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.091717 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090966    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-host-cni-netd\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.091717 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.090991    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vqjd\" (UniqueName: \"kubernetes.io/projected/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-kube-api-access-6vqjd\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.091717 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091003    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-cnibin\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:41.091717 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091014    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a202ee1a-5714-4706-9464-7076448f3bf0-etc-kubernetes\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.092441 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091055    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74d1b87b-73ed-4d75-b943-e2f8a13808be-tuning-conf-dir\") pod \"multus-additional-cni-plugins-njrdx\" (UID: \"74d1b87b-73ed-4d75-b943-e2f8a13808be\") " pod="openshift-multus/multus-additional-cni-plugins-njrdx"
Sep 16 17:24:41.092441 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091084    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxv2t\" (UniqueName: \"kubernetes.io/projected/0e392d83-bd3a-4d0b-ac4f-300a6019c434-kube-api-access-rxv2t\") pod \"network-metrics-daemon-shvlb\" (UID: \"0e392d83-bd3a-4d0b-ac4f-300a6019c434\") " pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:24:41.092441 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091110    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c910d08c-80cd-486a-98b3-518d769d149f-konnectivity-ca\") pod \"konnectivity-agent-9k478\" (UID: \"c910d08c-80cd-486a-98b3-518d769d149f\") " pod="kube-system/konnectivity-agent-9k478"
Sep 16 17:24:41.092441 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091131    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-system-cni-dir\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:41.092441 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091172    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-host-var-lib-cni-bin\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:41.092441 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091198    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcqhb\" (UniqueName: \"kubernetes.io/projected/a202ee1a-5714-4706-9464-7076448f3bf0-kube-api-access-pcqhb\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.092441 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091205    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74d1b87b-73ed-4d75-b943-e2f8a13808be-tuning-conf-dir\") pod \"multus-additional-cni-plugins-njrdx\" (UID: \"74d1b87b-73ed-4d75-b943-e2f8a13808be\") " pod="openshift-multus/multus-additional-cni-plugins-njrdx"
Sep 16 17:24:41.092441 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091249    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/451de579-98aa-48be-aeee-cf7d09759d3e-original-pull-secret\") pod \"global-pull-secret-syncer-6dj5q\" (UID: \"451de579-98aa-48be-aeee-cf7d09759d3e\") " pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:24:41.092441 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091281    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-host-run-multus-certs\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:41.092441 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091290    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-host-var-lib-cni-bin\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:41.092441 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091302    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-run-systemd\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.092441 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091355    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-host-run-multus-certs\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:41.092441 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091440    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 17:24:41.092441 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091449    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-host-cni-bin\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.092441 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091445    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-system-cni-dir\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:41.092441 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091474    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/74d1b87b-73ed-4d75-b943-e2f8a13808be-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-njrdx\" (UID: \"74d1b87b-73ed-4d75-b943-e2f8a13808be\") " pod="openshift-multus/multus-additional-cni-plugins-njrdx"
Sep 16 17:24:41.092441 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091494    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4c1797a6-6455-460e-a992-8bcd6171acff-registration-dir\") pod \"aws-ebs-csi-driver-node-ms5pm\" (UID: \"4c1797a6-6455-460e-a992-8bcd6171acff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms5pm"
Sep 16 17:24:41.093144 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091526    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ffd13091-dadb-4b9f-bd59-53f1073acedf-serviceca\") pod \"node-ca-dwmgh\" (UID: \"ffd13091-dadb-4b9f-bd59-53f1073acedf\") " pod="openshift-image-registry/node-ca-dwmgh"
Sep 16 17:24:41.093144 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091550    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/451de579-98aa-48be-aeee-cf7d09759d3e-kubelet-config\") pod \"global-pull-secret-syncer-6dj5q\" (UID: \"451de579-98aa-48be-aeee-cf7d09759d3e\") " pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:24:41.093144 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091580    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b674965e-c124-4b4c-ae07-0e17f355f0e0-cni-binary-copy\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:41.093144 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091614    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-hostroot\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:41.093144 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091672    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-hostroot\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:41.093144 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091718    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b674965e-c124-4b4c-ae07-0e17f355f0e0-multus-daemon-config\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:41.093144 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091734    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c910d08c-80cd-486a-98b3-518d769d149f-konnectivity-ca\") pod \"konnectivity-agent-9k478\" (UID: \"c910d08c-80cd-486a-98b3-518d769d149f\") " pod="kube-system/konnectivity-agent-9k478"
Sep 16 17:24:41.093144 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091758    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-ovnkube-script-lib\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.093144 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091784    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4c1797a6-6455-460e-a992-8bcd6171acff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ms5pm\" (UID: \"4c1797a6-6455-460e-a992-8bcd6171acff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms5pm"
Sep 16 17:24:41.093144 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-host-var-lib-kubelet\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:41.093144 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091885    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.093144 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091918    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a202ee1a-5714-4706-9464-7076448f3bf0-etc-modprobe-d\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.093144 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091945    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a202ee1a-5714-4706-9464-7076448f3bf0-etc-sysconfig\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.093144 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091960    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b674965e-c124-4b4c-ae07-0e17f355f0e0-host-var-lib-kubelet\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:41.093144 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091969    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a202ee1a-5714-4706-9464-7076448f3bf0-etc-systemd\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.093144 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.091999    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a202ee1a-5714-4706-9464-7076448f3bf0-host\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.093144 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.092039    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a202ee1a-5714-4706-9464-7076448f3bf0-etc-tuned\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.093746 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.092065    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4c1797a6-6455-460e-a992-8bcd6171acff-sys-fs\") pod \"aws-ebs-csi-driver-node-ms5pm\" (UID: \"4c1797a6-6455-460e-a992-8bcd6171acff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms5pm"
Sep 16 17:24:41.093746 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.092123    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/74d1b87b-73ed-4d75-b943-e2f8a13808be-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-njrdx\" (UID: \"74d1b87b-73ed-4d75-b943-e2f8a13808be\") " pod="openshift-multus/multus-additional-cni-plugins-njrdx"
Sep 16 17:24:41.093746 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.092153    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-host-kubelet\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.093746 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.092178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfqb8\" (UniqueName: \"kubernetes.io/projected/ffd13091-dadb-4b9f-bd59-53f1073acedf-kube-api-access-zfqb8\") pod \"node-ca-dwmgh\" (UID: \"ffd13091-dadb-4b9f-bd59-53f1073acedf\") " pod="openshift-image-registry/node-ca-dwmgh"
Sep 16 17:24:41.093746 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.092205    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmlq7\" (UniqueName: \"kubernetes.io/projected/b674965e-c124-4b4c-ae07-0e17f355f0e0-kube-api-access-jmlq7\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:41.093746 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.092280    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b674965e-c124-4b4c-ae07-0e17f355f0e0-multus-daemon-config\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:41.093746 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.092287    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b674965e-c124-4b4c-ae07-0e17f355f0e0-cni-binary-copy\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:41.093746 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.092646    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/74d1b87b-73ed-4d75-b943-e2f8a13808be-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-njrdx\" (UID: \"74d1b87b-73ed-4d75-b943-e2f8a13808be\") " pod="openshift-multus/multus-additional-cni-plugins-njrdx"
Sep 16 17:24:41.095504 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.095484    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c910d08c-80cd-486a-98b3-518d769d149f-agent-certs\") pod \"konnectivity-agent-9k478\" (UID: \"c910d08c-80cd-486a-98b3-518d769d149f\") " pod="kube-system/konnectivity-agent-9k478"
Sep 16 17:24:41.097057 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:41.097038    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:24:41.097146 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:41.097059    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:24:41.097146 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:41.097073    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jd4c6 for pod openshift-network-diagnostics/network-check-target-wklqk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:41.097146 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:41.097130    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d5f021d-6241-485a-8611-9fd0beb2ac98-kube-api-access-jd4c6 podName:5d5f021d-6241-485a-8611-9fd0beb2ac98 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:41.597113498 +0000 UTC m=+2.959792281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jd4c6" (UniqueName: "kubernetes.io/projected/5d5f021d-6241-485a-8611-9fd0beb2ac98-kube-api-access-jd4c6") pod "network-check-target-wklqk" (UID: "5d5f021d-6241-485a-8611-9fd0beb2ac98") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:41.099547 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.099524    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj2t8\" (UniqueName: \"kubernetes.io/projected/74d1b87b-73ed-4d75-b943-e2f8a13808be-kube-api-access-cj2t8\") pod \"multus-additional-cni-plugins-njrdx\" (UID: \"74d1b87b-73ed-4d75-b943-e2f8a13808be\") " pod="openshift-multus/multus-additional-cni-plugins-njrdx"
Sep 16 17:24:41.102759 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.102739    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxv2t\" (UniqueName: \"kubernetes.io/projected/0e392d83-bd3a-4d0b-ac4f-300a6019c434-kube-api-access-rxv2t\") pod \"network-metrics-daemon-shvlb\" (UID: \"0e392d83-bd3a-4d0b-ac4f-300a6019c434\") " pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:24:41.109604 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.109586    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmlq7\" (UniqueName: \"kubernetes.io/projected/b674965e-c124-4b4c-ae07-0e17f355f0e0-kube-api-access-jmlq7\") pod \"multus-7rnft\" (UID: \"b674965e-c124-4b4c-ae07-0e17f355f0e0\") " pod="openshift-multus/multus-7rnft"
Sep 16 17:24:41.192579 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.192542    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-ovnkube-config\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.192579 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.192583    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-systemd-units\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.192825 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.192608    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-host-cni-netd\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.192825 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.192661    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-host-cni-netd\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.192825 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.192683    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-systemd-units\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.192825 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.192792    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vqjd\" (UniqueName: \"kubernetes.io/projected/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-kube-api-access-6vqjd\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.193084 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.192827    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a202ee1a-5714-4706-9464-7076448f3bf0-etc-kubernetes\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.193084 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.192858    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcqhb\" (UniqueName: \"kubernetes.io/projected/a202ee1a-5714-4706-9464-7076448f3bf0-kube-api-access-pcqhb\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.193084 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.192899    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/451de579-98aa-48be-aeee-cf7d09759d3e-original-pull-secret\") pod \"global-pull-secret-syncer-6dj5q\" (UID: \"451de579-98aa-48be-aeee-cf7d09759d3e\") " pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:24:41.193084 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.192927    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-run-systemd\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.193084 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.192944    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a202ee1a-5714-4706-9464-7076448f3bf0-etc-kubernetes\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.193084 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.192952    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-host-cni-bin\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.193084 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193007    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-run-systemd\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.193084 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193006    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-host-cni-bin\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.193084 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193044    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4c1797a6-6455-460e-a992-8bcd6171acff-registration-dir\") pod \"aws-ebs-csi-driver-node-ms5pm\" (UID: \"4c1797a6-6455-460e-a992-8bcd6171acff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms5pm"
Sep 16 17:24:41.193084 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193077    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ffd13091-dadb-4b9f-bd59-53f1073acedf-serviceca\") pod \"node-ca-dwmgh\" (UID: \"ffd13091-dadb-4b9f-bd59-53f1073acedf\") " pod="openshift-image-registry/node-ca-dwmgh"
Sep 16 17:24:41.193084 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:41.193080    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:41.193672 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193128    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4c1797a6-6455-460e-a992-8bcd6171acff-registration-dir\") pod \"aws-ebs-csi-driver-node-ms5pm\" (UID: \"4c1797a6-6455-460e-a992-8bcd6171acff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms5pm"
Sep 16 17:24:41.193672 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193159    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/451de579-98aa-48be-aeee-cf7d09759d3e-kubelet-config\") pod \"global-pull-secret-syncer-6dj5q\" (UID: \"451de579-98aa-48be-aeee-cf7d09759d3e\") " pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:24:41.193672 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193106    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/451de579-98aa-48be-aeee-cf7d09759d3e-kubelet-config\") pod \"global-pull-secret-syncer-6dj5q\" (UID: \"451de579-98aa-48be-aeee-cf7d09759d3e\") " pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:24:41.193672 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:41.193182    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/451de579-98aa-48be-aeee-cf7d09759d3e-original-pull-secret podName:451de579-98aa-48be-aeee-cf7d09759d3e nodeName:}" failed. No retries permitted until 2026-09-16 17:24:41.693163359 +0000 UTC m=+3.055842128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/451de579-98aa-48be-aeee-cf7d09759d3e-original-pull-secret") pod "global-pull-secret-syncer-6dj5q" (UID: "451de579-98aa-48be-aeee-cf7d09759d3e") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:41.193672 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193223    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-ovnkube-script-lib\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.193672 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193255    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4c1797a6-6455-460e-a992-8bcd6171acff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ms5pm\" (UID: \"4c1797a6-6455-460e-a992-8bcd6171acff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms5pm"
Sep 16 17:24:41.193672 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193286    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.193672 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193313    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a202ee1a-5714-4706-9464-7076448f3bf0-etc-modprobe-d\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.193672 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193338    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a202ee1a-5714-4706-9464-7076448f3bf0-etc-sysconfig\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.193672 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193357    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-ovnkube-config\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.193672 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193375    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.193672 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193363    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a202ee1a-5714-4706-9464-7076448f3bf0-etc-systemd\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.193672 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193409    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a202ee1a-5714-4706-9464-7076448f3bf0-etc-systemd\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.193672 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193417    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a202ee1a-5714-4706-9464-7076448f3bf0-host\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.193672 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193337    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4c1797a6-6455-460e-a992-8bcd6171acff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ms5pm\" (UID: \"4c1797a6-6455-460e-a992-8bcd6171acff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms5pm"
Sep 16 17:24:41.193672 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193433    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a202ee1a-5714-4706-9464-7076448f3bf0-etc-sysconfig\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.194572 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193447    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a202ee1a-5714-4706-9464-7076448f3bf0-etc-tuned\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.194572 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193452    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a202ee1a-5714-4706-9464-7076448f3bf0-etc-modprobe-d\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.194572 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193475    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a202ee1a-5714-4706-9464-7076448f3bf0-host\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.194572 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193474    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4c1797a6-6455-460e-a992-8bcd6171acff-sys-fs\") pod \"aws-ebs-csi-driver-node-ms5pm\" (UID: \"4c1797a6-6455-460e-a992-8bcd6171acff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms5pm"
Sep 16 17:24:41.194572 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-host-kubelet\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.194572 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193545    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfqb8\" (UniqueName: \"kubernetes.io/projected/ffd13091-dadb-4b9f-bd59-53f1073acedf-kube-api-access-zfqb8\") pod \"node-ca-dwmgh\" (UID: \"ffd13091-dadb-4b9f-bd59-53f1073acedf\") " pod="openshift-image-registry/node-ca-dwmgh"
Sep 16 17:24:41.194572 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193572    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/445899b8-af25-4084-a9c8-d917110aea4b-iptables-alerter-script\") pod \"iptables-alerter-j2xzd\" (UID: \"445899b8-af25-4084-a9c8-d917110aea4b\") " pod="openshift-network-operator/iptables-alerter-j2xzd"
Sep 16 17:24:41.194572 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193576    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ffd13091-dadb-4b9f-bd59-53f1073acedf-serviceca\") pod \"node-ca-dwmgh\" (UID: \"ffd13091-dadb-4b9f-bd59-53f1073acedf\") " pod="openshift-image-registry/node-ca-dwmgh"
Sep 16 17:24:41.194572 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193596    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/445899b8-af25-4084-a9c8-d917110aea4b-host-slash\") pod \"iptables-alerter-j2xzd\" (UID: \"445899b8-af25-4084-a9c8-d917110aea4b\") " pod="openshift-network-operator/iptables-alerter-j2xzd"
Sep 16 17:24:41.194572 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193627    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4c1797a6-6455-460e-a992-8bcd6171acff-sys-fs\") pod \"aws-ebs-csi-driver-node-ms5pm\" (UID: \"4c1797a6-6455-460e-a992-8bcd6171acff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms5pm"
Sep 16 17:24:41.194572 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193632    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-host-kubelet\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.194572 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193621    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-host-run-netns\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.194572 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193666    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-host-run-netns\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.194572 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193676    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-host-run-ovn-kubernetes\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.194572 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193704    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/445899b8-af25-4084-a9c8-d917110aea4b-host-slash\") pod \"iptables-alerter-j2xzd\" (UID: \"445899b8-af25-4084-a9c8-d917110aea4b\") " pod="openshift-network-operator/iptables-alerter-j2xzd"
Sep 16 17:24:41.194572 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193717    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xr22\" (UniqueName: \"kubernetes.io/projected/445899b8-af25-4084-a9c8-d917110aea4b-kube-api-access-5xr22\") pod \"iptables-alerter-j2xzd\" (UID: \"445899b8-af25-4084-a9c8-d917110aea4b\") " pod="openshift-network-operator/iptables-alerter-j2xzd"
Sep 16 17:24:41.194572 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193742    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-var-lib-openvswitch\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.195387 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193749    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-host-run-ovn-kubernetes\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.195387 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193766    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-node-log\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.195387 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193792    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a202ee1a-5714-4706-9464-7076448f3bf0-sys\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.195387 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193796    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-ovnkube-script-lib\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.195387 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193807    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-var-lib-openvswitch\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.195387 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193839    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a202ee1a-5714-4706-9464-7076448f3bf0-etc-sysctl-d\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.195387 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193860    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a202ee1a-5714-4706-9464-7076448f3bf0-sys\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.195387 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193868    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwrpw\" (UniqueName: \"kubernetes.io/projected/4c1797a6-6455-460e-a992-8bcd6171acff-kube-api-access-lwrpw\") pod \"aws-ebs-csi-driver-node-ms5pm\" (UID: \"4c1797a6-6455-460e-a992-8bcd6171acff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms5pm"
Sep 16 17:24:41.195387 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193870    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-node-log\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.195387 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193896    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffd13091-dadb-4b9f-bd59-53f1073acedf-host\") pod \"node-ca-dwmgh\" (UID: \"ffd13091-dadb-4b9f-bd59-53f1073acedf\") " pod="openshift-image-registry/node-ca-dwmgh"
Sep 16 17:24:41.195387 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193941    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffd13091-dadb-4b9f-bd59-53f1073acedf-host\") pod \"node-ca-dwmgh\" (UID: \"ffd13091-dadb-4b9f-bd59-53f1073acedf\") " pod="openshift-image-registry/node-ca-dwmgh"
Sep 16 17:24:41.195387 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193939    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-log-socket\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.195387 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193981    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-ovn-node-metrics-cert\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.195387 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.193987    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-log-socket\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.195387 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.194007    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a202ee1a-5714-4706-9464-7076448f3bf0-etc-sysctl-d\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.195387 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.194013    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4c1797a6-6455-460e-a992-8bcd6171acff-socket-dir\") pod \"aws-ebs-csi-driver-node-ms5pm\" (UID: \"4c1797a6-6455-460e-a992-8bcd6171acff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms5pm"
Sep 16 17:24:41.195387 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.194042    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2r7g\" (UniqueName: \"kubernetes.io/projected/c6e2238e-ef6c-463e-8123-4663b5961709-kube-api-access-m2r7g\") pod \"node-resolver-jfsjw\" (UID: \"c6e2238e-ef6c-463e-8123-4663b5961709\") " pod="openshift-dns/node-resolver-jfsjw"
Sep 16 17:24:41.195387 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.194077    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-host-slash\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.196121 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.194102    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/451de579-98aa-48be-aeee-cf7d09759d3e-dbus\") pod \"global-pull-secret-syncer-6dj5q\" (UID: \"451de579-98aa-48be-aeee-cf7d09759d3e\") " pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:24:41.196121 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.194127    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-etc-openvswitch\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.196121 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.194151    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-run-openvswitch\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.196121 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.194174    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4c1797a6-6455-460e-a992-8bcd6171acff-socket-dir\") pod \"aws-ebs-csi-driver-node-ms5pm\" (UID: \"4c1797a6-6455-460e-a992-8bcd6171acff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms5pm"
Sep 16 17:24:41.196121 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.194217    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-run-ovn\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.196121 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.194214    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/451de579-98aa-48be-aeee-cf7d09759d3e-dbus\") pod \"global-pull-secret-syncer-6dj5q\" (UID: \"451de579-98aa-48be-aeee-cf7d09759d3e\") " pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:24:41.196121 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.194233    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-etc-openvswitch\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.196121 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.194244    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a202ee1a-5714-4706-9464-7076448f3bf0-run\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.196121 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.194270    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a202ee1a-5714-4706-9464-7076448f3bf0-lib-modules\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.196121 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.194283    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-host-slash\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.196121 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.194288    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-run-openvswitch\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.196121 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.194303    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a202ee1a-5714-4706-9464-7076448f3bf0-var-lib-kubelet\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.196121 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.194330    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a202ee1a-5714-4706-9464-7076448f3bf0-run\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.196121 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.194335    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-run-ovn\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.196121 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.194329    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6e2238e-ef6c-463e-8123-4663b5961709-tmp-dir\") pod \"node-resolver-jfsjw\" (UID: \"c6e2238e-ef6c-463e-8123-4663b5961709\") " pod="openshift-dns/node-resolver-jfsjw"
Sep 16 17:24:41.196121 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.194378    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-env-overrides\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.196121 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.194391    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a202ee1a-5714-4706-9464-7076448f3bf0-lib-modules\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.196121 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.194394    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a202ee1a-5714-4706-9464-7076448f3bf0-var-lib-kubelet\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.196977 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.194407    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a202ee1a-5714-4706-9464-7076448f3bf0-tmp\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.196977 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.194433    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4c1797a6-6455-460e-a992-8bcd6171acff-device-dir\") pod \"aws-ebs-csi-driver-node-ms5pm\" (UID: \"4c1797a6-6455-460e-a992-8bcd6171acff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms5pm"
Sep 16 17:24:41.196977 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.194459    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c6e2238e-ef6c-463e-8123-4663b5961709-hosts-file\") pod \"node-resolver-jfsjw\" (UID: \"c6e2238e-ef6c-463e-8123-4663b5961709\") " pod="openshift-dns/node-resolver-jfsjw"
Sep 16 17:24:41.196977 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.194486    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a202ee1a-5714-4706-9464-7076448f3bf0-etc-sysctl-conf\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.196977 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.194513    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4c1797a6-6455-460e-a992-8bcd6171acff-etc-selinux\") pod \"aws-ebs-csi-driver-node-ms5pm\" (UID: \"4c1797a6-6455-460e-a992-8bcd6171acff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms5pm"
Sep 16 17:24:41.196977 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.194518    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4c1797a6-6455-460e-a992-8bcd6171acff-device-dir\") pod \"aws-ebs-csi-driver-node-ms5pm\" (UID: \"4c1797a6-6455-460e-a992-8bcd6171acff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms5pm"
Sep 16 17:24:41.196977 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.194529    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c6e2238e-ef6c-463e-8123-4663b5961709-hosts-file\") pod \"node-resolver-jfsjw\" (UID: \"c6e2238e-ef6c-463e-8123-4663b5961709\") " pod="openshift-dns/node-resolver-jfsjw"
Sep 16 17:24:41.196977 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.194602    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4c1797a6-6455-460e-a992-8bcd6171acff-etc-selinux\") pod \"aws-ebs-csi-driver-node-ms5pm\" (UID: \"4c1797a6-6455-460e-a992-8bcd6171acff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms5pm"
Sep 16 17:24:41.196977 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.194659    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6e2238e-ef6c-463e-8123-4663b5961709-tmp-dir\") pod \"node-resolver-jfsjw\" (UID: \"c6e2238e-ef6c-463e-8123-4663b5961709\") " pod="openshift-dns/node-resolver-jfsjw"
Sep 16 17:24:41.196977 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.194675    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a202ee1a-5714-4706-9464-7076448f3bf0-etc-sysctl-conf\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.196977 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.194810    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-env-overrides\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.196977 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.195172    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/445899b8-af25-4084-a9c8-d917110aea4b-iptables-alerter-script\") pod \"iptables-alerter-j2xzd\" (UID: \"445899b8-af25-4084-a9c8-d917110aea4b\") " pod="openshift-network-operator/iptables-alerter-j2xzd"
Sep 16 17:24:41.196977 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.196325    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a202ee1a-5714-4706-9464-7076448f3bf0-etc-tuned\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.196977 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.196765    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-ovn-node-metrics-cert\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.196977 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.196973    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a202ee1a-5714-4706-9464-7076448f3bf0-tmp\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.208636 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.208599    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcqhb\" (UniqueName: \"kubernetes.io/projected/a202ee1a-5714-4706-9464-7076448f3bf0-kube-api-access-pcqhb\") pod \"tuned-jdgc7\" (UID: \"a202ee1a-5714-4706-9464-7076448f3bf0\") " pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.208636 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.208610    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vqjd\" (UniqueName: \"kubernetes.io/projected/3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0-kube-api-access-6vqjd\") pod \"ovnkube-node-5btzs\" (UID: \"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.211489 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.211427    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xr22\" (UniqueName: \"kubernetes.io/projected/445899b8-af25-4084-a9c8-d917110aea4b-kube-api-access-5xr22\") pod \"iptables-alerter-j2xzd\" (UID: \"445899b8-af25-4084-a9c8-d917110aea4b\") " pod="openshift-network-operator/iptables-alerter-j2xzd"
Sep 16 17:24:41.211489 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.211434    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwrpw\" (UniqueName: \"kubernetes.io/projected/4c1797a6-6455-460e-a992-8bcd6171acff-kube-api-access-lwrpw\") pod \"aws-ebs-csi-driver-node-ms5pm\" (UID: \"4c1797a6-6455-460e-a992-8bcd6171acff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms5pm"
Sep 16 17:24:41.211665 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.211595    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfqb8\" (UniqueName: \"kubernetes.io/projected/ffd13091-dadb-4b9f-bd59-53f1073acedf-kube-api-access-zfqb8\") pod \"node-ca-dwmgh\" (UID: \"ffd13091-dadb-4b9f-bd59-53f1073acedf\") " pod="openshift-image-registry/node-ca-dwmgh"
Sep 16 17:24:41.211665 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.211594    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2r7g\" (UniqueName: \"kubernetes.io/projected/c6e2238e-ef6c-463e-8123-4663b5961709-kube-api-access-m2r7g\") pod \"node-resolver-jfsjw\" (UID: \"c6e2238e-ef6c-463e-8123-4663b5961709\") " pod="openshift-dns/node-resolver-jfsjw"
Sep 16 17:24:41.233203 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.233180    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 17:24:41.287060 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.287019    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9k478"
Sep 16 17:24:41.296229 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.296205    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7rnft"
Sep 16 17:24:41.308255 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.308231    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-njrdx"
Sep 16 17:24:41.313293 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.313266    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j2xzd"
Sep 16 17:24:41.323304 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.323280    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms5pm"
Sep 16 17:24:41.332329 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.332301    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jfsjw"
Sep 16 17:24:41.341439 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.341406    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:24:41.351377 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.351357    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jdgc7"
Sep 16 17:24:41.357005 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.356983    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dwmgh"
Sep 16 17:24:41.598057 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.597969    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e392d83-bd3a-4d0b-ac4f-300a6019c434-metrics-certs\") pod \"network-metrics-daemon-shvlb\" (UID: \"0e392d83-bd3a-4d0b-ac4f-300a6019c434\") " pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:24:41.598057 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.598022    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd4c6\" (UniqueName: \"kubernetes.io/projected/5d5f021d-6241-485a-8611-9fd0beb2ac98-kube-api-access-jd4c6\") pod \"network-check-target-wklqk\" (UID: \"5d5f021d-6241-485a-8611-9fd0beb2ac98\") " pod="openshift-network-diagnostics/network-check-target-wklqk"
Sep 16 17:24:41.598268 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:41.598154    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:41.598268 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:41.598223    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e392d83-bd3a-4d0b-ac4f-300a6019c434-metrics-certs podName:0e392d83-bd3a-4d0b-ac4f-300a6019c434 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:42.59820588 +0000 UTC m=+3.960884661 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e392d83-bd3a-4d0b-ac4f-300a6019c434-metrics-certs") pod "network-metrics-daemon-shvlb" (UID: "0e392d83-bd3a-4d0b-ac4f-300a6019c434") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:41.598268 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:41.598165    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:24:41.598268 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:41.598270    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:24:41.598458 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:41.598288    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jd4c6 for pod openshift-network-diagnostics/network-check-target-wklqk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:41.598458 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:41.598353    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d5f021d-6241-485a-8611-9fd0beb2ac98-kube-api-access-jd4c6 podName:5d5f021d-6241-485a-8611-9fd0beb2ac98 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:42.598333536 +0000 UTC m=+3.961012316 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jd4c6" (UniqueName: "kubernetes.io/projected/5d5f021d-6241-485a-8611-9fd0beb2ac98-kube-api-access-jd4c6") pod "network-check-target-wklqk" (UID: "5d5f021d-6241-485a-8611-9fd0beb2ac98") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:41.698693 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:41.698655    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/451de579-98aa-48be-aeee-cf7d09759d3e-original-pull-secret\") pod \"global-pull-secret-syncer-6dj5q\" (UID: \"451de579-98aa-48be-aeee-cf7d09759d3e\") " pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:24:41.698860 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:41.698783    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:41.698860 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:41.698849    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/451de579-98aa-48be-aeee-cf7d09759d3e-original-pull-secret podName:451de579-98aa-48be-aeee-cf7d09759d3e nodeName:}" failed. No retries permitted until 2026-09-16 17:24:42.698832832 +0000 UTC m=+4.061511607 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/451de579-98aa-48be-aeee-cf7d09759d3e-original-pull-secret") pod "global-pull-secret-syncer-6dj5q" (UID: "451de579-98aa-48be-aeee-cf7d09759d3e") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:42.057882 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:42.057788    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 17:19:41 +0000 UTC" deadline="2028-05-07 09:29:20.261971431 +0000 UTC"
Sep 16 17:24:42.057882 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:42.057837    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14368h4m38.204139696s"
Sep 16 17:24:42.074173 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:42.074139    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wklqk"
Sep 16 17:24:42.074307 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:42.074261    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wklqk" podUID="5d5f021d-6241-485a-8611-9fd0beb2ac98"
Sep 16 17:24:42.148881 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:42.148763    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc910d08c_80cd_486a_98b3_518d769d149f.slice/crio-37801f990ceea29295150b57f4e530151a93f55a202cce137f4c0e07c5497246 WatchSource:0}: Error finding container 37801f990ceea29295150b57f4e530151a93f55a202cce137f4c0e07c5497246: Status 404 returned error can't find the container with id 37801f990ceea29295150b57f4e530151a93f55a202cce137f4c0e07c5497246
Sep 16 17:24:42.150718 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:42.150692    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74d1b87b_73ed_4d75_b943_e2f8a13808be.slice/crio-7c614e9a3ff6941564737302a7062c9a2d5cf08cf6aee0ea30d757ad08395b9b WatchSource:0}: Error finding container 7c614e9a3ff6941564737302a7062c9a2d5cf08cf6aee0ea30d757ad08395b9b: Status 404 returned error can't find the container with id 7c614e9a3ff6941564737302a7062c9a2d5cf08cf6aee0ea30d757ad08395b9b
Sep 16 17:24:42.151598 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:42.151421    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e2238e_ef6c_463e_8123_4663b5961709.slice/crio-3e5c6876ba75df29b0f75339361e28a00af414fe58d2ec542249413061bc4fe7 WatchSource:0}: Error finding container 3e5c6876ba75df29b0f75339361e28a00af414fe58d2ec542249413061bc4fe7: Status 404 returned error can't find the container with id 3e5c6876ba75df29b0f75339361e28a00af414fe58d2ec542249413061bc4fe7
Sep 16 17:24:42.155502 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:42.155283    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda202ee1a_5714_4706_9464_7076448f3bf0.slice/crio-53dde6aa72986b81faa2402060214fd2b6314dd83f9b326ac15d3773c39164a7 WatchSource:0}: Error finding container 53dde6aa72986b81faa2402060214fd2b6314dd83f9b326ac15d3773c39164a7: Status 404 returned error can't find the container with id 53dde6aa72986b81faa2402060214fd2b6314dd83f9b326ac15d3773c39164a7
Sep 16 17:24:42.156026 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:42.155998    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod445899b8_af25_4084_a9c8_d917110aea4b.slice/crio-184cd86b2295195187aa4167af69645d62d550f045b00a45ec793d90888af72e WatchSource:0}: Error finding container 184cd86b2295195187aa4167af69645d62d550f045b00a45ec793d90888af72e: Status 404 returned error can't find the container with id 184cd86b2295195187aa4167af69645d62d550f045b00a45ec793d90888af72e
Sep 16 17:24:42.157078 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:42.157054    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd13091_dadb_4b9f_bd59_53f1073acedf.slice/crio-4a1c9514bc3933f4561bc1a5ac439c2894661dbc5bba2647166c87f72d5042a9 WatchSource:0}: Error finding container 4a1c9514bc3933f4561bc1a5ac439c2894661dbc5bba2647166c87f72d5042a9: Status 404 returned error can't find the container with id 4a1c9514bc3933f4561bc1a5ac439c2894661dbc5bba2647166c87f72d5042a9
Sep 16 17:24:42.158660 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:42.158151    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb674965e_c124_4b4c_ae07_0e17f355f0e0.slice/crio-a36b9429b30c0165c4f8e0de7a3a7c53daed668011fe77cd09b70afda34935bd WatchSource:0}: Error finding container a36b9429b30c0165c4f8e0de7a3a7c53daed668011fe77cd09b70afda34935bd: Status 404 returned error can't find the container with id a36b9429b30c0165c4f8e0de7a3a7c53daed668011fe77cd09b70afda34935bd
Sep 16 17:24:42.159991 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:42.159970    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3909b38e_18eb_4b4a_8ec0_0ccb1d6b81c0.slice/crio-f696aacae5b9e233589c7f8dc1a1a1fba02cf6b8638a7e03238472804c81b505 WatchSource:0}: Error finding container f696aacae5b9e233589c7f8dc1a1a1fba02cf6b8638a7e03238472804c81b505: Status 404 returned error can't find the container with id f696aacae5b9e233589c7f8dc1a1a1fba02cf6b8638a7e03238472804c81b505
Sep 16 17:24:42.160365 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:24:42.160320    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c1797a6_6455_460e_a992_8bcd6171acff.slice/crio-20e9641d9f97585667b3e259af630f45acb1149c71ca0056af34a1e4c97eceb5 WatchSource:0}: Error finding container 20e9641d9f97585667b3e259af630f45acb1149c71ca0056af34a1e4c97eceb5: Status 404 returned error can't find the container with id 20e9641d9f97585667b3e259af630f45acb1149c71ca0056af34a1e4c97eceb5
Sep 16 17:24:42.607331 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:42.606338    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd4c6\" (UniqueName: \"kubernetes.io/projected/5d5f021d-6241-485a-8611-9fd0beb2ac98-kube-api-access-jd4c6\") pod \"network-check-target-wklqk\" (UID: \"5d5f021d-6241-485a-8611-9fd0beb2ac98\") " pod="openshift-network-diagnostics/network-check-target-wklqk"
Sep 16 17:24:42.607331 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:42.606449    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e392d83-bd3a-4d0b-ac4f-300a6019c434-metrics-certs\") pod \"network-metrics-daemon-shvlb\" (UID: \"0e392d83-bd3a-4d0b-ac4f-300a6019c434\") " pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:24:42.607331 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:42.606581    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:42.607331 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:42.606653    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e392d83-bd3a-4d0b-ac4f-300a6019c434-metrics-certs podName:0e392d83-bd3a-4d0b-ac4f-300a6019c434 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:44.606632612 +0000 UTC m=+5.969311392 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e392d83-bd3a-4d0b-ac4f-300a6019c434-metrics-certs") pod "network-metrics-daemon-shvlb" (UID: "0e392d83-bd3a-4d0b-ac4f-300a6019c434") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:42.607331 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:42.607191    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:24:42.607331 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:42.607217    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:24:42.607331 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:42.607231    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jd4c6 for pod openshift-network-diagnostics/network-check-target-wklqk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:42.607331 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:42.607283    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d5f021d-6241-485a-8611-9fd0beb2ac98-kube-api-access-jd4c6 podName:5d5f021d-6241-485a-8611-9fd0beb2ac98 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:44.607266623 +0000 UTC m=+5.969945404 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jd4c6" (UniqueName: "kubernetes.io/projected/5d5f021d-6241-485a-8611-9fd0beb2ac98-kube-api-access-jd4c6") pod "network-check-target-wklqk" (UID: "5d5f021d-6241-485a-8611-9fd0beb2ac98") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:42.707987 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:42.707244    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/451de579-98aa-48be-aeee-cf7d09759d3e-original-pull-secret\") pod \"global-pull-secret-syncer-6dj5q\" (UID: \"451de579-98aa-48be-aeee-cf7d09759d3e\") " pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:24:42.707987 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:42.707457    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:42.707987 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:42.707528    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/451de579-98aa-48be-aeee-cf7d09759d3e-original-pull-secret podName:451de579-98aa-48be-aeee-cf7d09759d3e nodeName:}" failed. No retries permitted until 2026-09-16 17:24:44.70750947 +0000 UTC m=+6.070188238 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/451de579-98aa-48be-aeee-cf7d09759d3e-original-pull-secret") pod "global-pull-secret-syncer-6dj5q" (UID: "451de579-98aa-48be-aeee-cf7d09759d3e") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:43.058749 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:43.058619    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 17:19:41 +0000 UTC" deadline="2028-06-01 19:26:11.28417599 +0000 UTC"
Sep 16 17:24:43.058749 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:43.058694    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14978h1m28.225487288s"
Sep 16 17:24:43.075225 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:43.075192    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:24:43.075395 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:43.075351    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dj5q" podUID="451de579-98aa-48be-aeee-cf7d09759d3e"
Sep 16 17:24:43.075889 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:43.075856    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:24:43.076013 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:43.075991    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shvlb" podUID="0e392d83-bd3a-4d0b-ac4f-300a6019c434"
Sep 16 17:24:43.116379 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:43.115439    2577 generic.go:356] "Generic (PLEG): container finished" podID="80b06e6b7e68304bfaedbddff98b161c" containerID="95b9e2665fe117aa2956b44971d1d63297b8086efd4e17cebfd6e5b28e4f6999" exitCode=0
Sep 16 17:24:43.116379 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:43.116140    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-122.ec2.internal" event={"ID":"80b06e6b7e68304bfaedbddff98b161c","Type":"ContainerDied","Data":"95b9e2665fe117aa2956b44971d1d63297b8086efd4e17cebfd6e5b28e4f6999"}
Sep 16 17:24:43.118594 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:43.118510    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j2xzd" event={"ID":"445899b8-af25-4084-a9c8-d917110aea4b","Type":"ContainerStarted","Data":"184cd86b2295195187aa4167af69645d62d550f045b00a45ec793d90888af72e"}
Sep 16 17:24:43.124674 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:43.124603    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms5pm" event={"ID":"4c1797a6-6455-460e-a992-8bcd6171acff","Type":"ContainerStarted","Data":"20e9641d9f97585667b3e259af630f45acb1149c71ca0056af34a1e4c97eceb5"}
Sep 16 17:24:43.131413 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:43.131329    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dwmgh" event={"ID":"ffd13091-dadb-4b9f-bd59-53f1073acedf","Type":"ContainerStarted","Data":"4a1c9514bc3933f4561bc1a5ac439c2894661dbc5bba2647166c87f72d5042a9"}
Sep 16 17:24:43.156847 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:43.156751    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njrdx" event={"ID":"74d1b87b-73ed-4d75-b943-e2f8a13808be","Type":"ContainerStarted","Data":"7c614e9a3ff6941564737302a7062c9a2d5cf08cf6aee0ea30d757ad08395b9b"}
Sep 16 17:24:43.187993 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:43.187077    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-122.ec2.internal" event={"ID":"b8e15c478aefab14292c7bbfa4653ed5","Type":"ContainerStarted","Data":"c6fa68b501e199308bd091ad0fbf3baf2dd2a41effa9d2a6ec5a39ba740b2b14"}
Sep 16 17:24:43.215999 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:43.215931    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5btzs" event={"ID":"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0","Type":"ContainerStarted","Data":"f696aacae5b9e233589c7f8dc1a1a1fba02cf6b8638a7e03238472804c81b505"}
Sep 16 17:24:43.232415 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:43.232382    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7rnft" event={"ID":"b674965e-c124-4b4c-ae07-0e17f355f0e0","Type":"ContainerStarted","Data":"a36b9429b30c0165c4f8e0de7a3a7c53daed668011fe77cd09b70afda34935bd"}
Sep 16 17:24:43.234923 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:43.234882    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jdgc7" event={"ID":"a202ee1a-5714-4706-9464-7076448f3bf0","Type":"ContainerStarted","Data":"53dde6aa72986b81faa2402060214fd2b6314dd83f9b326ac15d3773c39164a7"}
Sep 16 17:24:43.238282 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:43.238246    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jfsjw" event={"ID":"c6e2238e-ef6c-463e-8123-4663b5961709","Type":"ContainerStarted","Data":"3e5c6876ba75df29b0f75339361e28a00af414fe58d2ec542249413061bc4fe7"}
Sep 16 17:24:43.255929 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:43.255890    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9k478" event={"ID":"c910d08c-80cd-486a-98b3-518d769d149f","Type":"ContainerStarted","Data":"37801f990ceea29295150b57f4e530151a93f55a202cce137f4c0e07c5497246"}
Sep 16 17:24:44.074506 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:44.074471    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wklqk"
Sep 16 17:24:44.075328 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:44.074622    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wklqk" podUID="5d5f021d-6241-485a-8611-9fd0beb2ac98"
Sep 16 17:24:44.277643 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:44.277559    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-122.ec2.internal" event={"ID":"80b06e6b7e68304bfaedbddff98b161c","Type":"ContainerStarted","Data":"6ba6e3605337f6c8482a0c36f2091e0834d178bc394514e7f61faa212a3e9e2e"}
Sep 16 17:24:44.293666 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:44.293597    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-122.ec2.internal" podStartSLOduration=4.293574405 podStartE2EDuration="4.293574405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:24:44.292764527 +0000 UTC m=+5.655443317" watchObservedRunningTime="2026-09-16 17:24:44.293574405 +0000 UTC m=+5.656253196"
Sep 16 17:24:44.294756 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:44.294080    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-122.ec2.internal" podStartSLOduration=4.294063077 podStartE2EDuration="4.294063077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:24:43.20282945 +0000 UTC m=+4.565508242" watchObservedRunningTime="2026-09-16 17:24:44.294063077 +0000 UTC m=+5.656741867"
Sep 16 17:24:44.625345 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:44.625245    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e392d83-bd3a-4d0b-ac4f-300a6019c434-metrics-certs\") pod \"network-metrics-daemon-shvlb\" (UID: \"0e392d83-bd3a-4d0b-ac4f-300a6019c434\") " pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:24:44.625345 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:44.625318    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd4c6\" (UniqueName: \"kubernetes.io/projected/5d5f021d-6241-485a-8611-9fd0beb2ac98-kube-api-access-jd4c6\") pod \"network-check-target-wklqk\" (UID: \"5d5f021d-6241-485a-8611-9fd0beb2ac98\") " pod="openshift-network-diagnostics/network-check-target-wklqk"
Sep 16 17:24:44.625582 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:44.625466    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:24:44.625582 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:44.625486    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:24:44.625582 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:44.625500    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jd4c6 for pod openshift-network-diagnostics/network-check-target-wklqk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:44.625582 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:44.625562    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d5f021d-6241-485a-8611-9fd0beb2ac98-kube-api-access-jd4c6 podName:5d5f021d-6241-485a-8611-9fd0beb2ac98 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:48.625543444 +0000 UTC m=+9.988222215 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jd4c6" (UniqueName: "kubernetes.io/projected/5d5f021d-6241-485a-8611-9fd0beb2ac98-kube-api-access-jd4c6") pod "network-check-target-wklqk" (UID: "5d5f021d-6241-485a-8611-9fd0beb2ac98") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:44.626010 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:44.625991    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:44.626087 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:44.626046    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e392d83-bd3a-4d0b-ac4f-300a6019c434-metrics-certs podName:0e392d83-bd3a-4d0b-ac4f-300a6019c434 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:48.626031653 +0000 UTC m=+9.988710423 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e392d83-bd3a-4d0b-ac4f-300a6019c434-metrics-certs") pod "network-metrics-daemon-shvlb" (UID: "0e392d83-bd3a-4d0b-ac4f-300a6019c434") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:44.726139 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:44.726074    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/451de579-98aa-48be-aeee-cf7d09759d3e-original-pull-secret\") pod \"global-pull-secret-syncer-6dj5q\" (UID: \"451de579-98aa-48be-aeee-cf7d09759d3e\") " pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:24:44.726342 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:44.726256    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:44.726342 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:44.726331    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/451de579-98aa-48be-aeee-cf7d09759d3e-original-pull-secret podName:451de579-98aa-48be-aeee-cf7d09759d3e nodeName:}" failed. No retries permitted until 2026-09-16 17:24:48.72631271 +0000 UTC m=+10.088991479 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/451de579-98aa-48be-aeee-cf7d09759d3e-original-pull-secret") pod "global-pull-secret-syncer-6dj5q" (UID: "451de579-98aa-48be-aeee-cf7d09759d3e") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:45.078559 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:45.077667    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:24:45.078559 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:45.077820    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shvlb" podUID="0e392d83-bd3a-4d0b-ac4f-300a6019c434"
Sep 16 17:24:45.078559 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:45.078282    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:24:45.078559 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:45.078384    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dj5q" podUID="451de579-98aa-48be-aeee-cf7d09759d3e"
Sep 16 17:24:46.075233 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:46.075195    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wklqk"
Sep 16 17:24:46.075428 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:46.075342    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wklqk" podUID="5d5f021d-6241-485a-8611-9fd0beb2ac98"
Sep 16 17:24:47.074420 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:47.074351    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:24:47.074880 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:47.074501    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dj5q" podUID="451de579-98aa-48be-aeee-cf7d09759d3e"
Sep 16 17:24:47.075013 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:47.074892    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:24:47.075076 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:47.075015    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shvlb" podUID="0e392d83-bd3a-4d0b-ac4f-300a6019c434"
Sep 16 17:24:48.074277 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:48.074160    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wklqk"
Sep 16 17:24:48.074463 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:48.074328    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wklqk" podUID="5d5f021d-6241-485a-8611-9fd0beb2ac98"
Sep 16 17:24:48.661327 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:48.661286    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e392d83-bd3a-4d0b-ac4f-300a6019c434-metrics-certs\") pod \"network-metrics-daemon-shvlb\" (UID: \"0e392d83-bd3a-4d0b-ac4f-300a6019c434\") " pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:24:48.661514 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:48.661350    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd4c6\" (UniqueName: \"kubernetes.io/projected/5d5f021d-6241-485a-8611-9fd0beb2ac98-kube-api-access-jd4c6\") pod \"network-check-target-wklqk\" (UID: \"5d5f021d-6241-485a-8611-9fd0beb2ac98\") " pod="openshift-network-diagnostics/network-check-target-wklqk"
Sep 16 17:24:48.661514 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:48.661459    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:48.661692 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:48.661542    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:24:48.661692 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:48.661560    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e392d83-bd3a-4d0b-ac4f-300a6019c434-metrics-certs podName:0e392d83-bd3a-4d0b-ac4f-300a6019c434 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:56.661534999 +0000 UTC m=+18.024213780 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e392d83-bd3a-4d0b-ac4f-300a6019c434-metrics-certs") pod "network-metrics-daemon-shvlb" (UID: "0e392d83-bd3a-4d0b-ac4f-300a6019c434") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:48.661692 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:48.661567    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:24:48.661692 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:48.661584    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jd4c6 for pod openshift-network-diagnostics/network-check-target-wklqk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:48.661692 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:48.661634    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d5f021d-6241-485a-8611-9fd0beb2ac98-kube-api-access-jd4c6 podName:5d5f021d-6241-485a-8611-9fd0beb2ac98 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:56.661619074 +0000 UTC m=+18.024297840 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jd4c6" (UniqueName: "kubernetes.io/projected/5d5f021d-6241-485a-8611-9fd0beb2ac98-kube-api-access-jd4c6") pod "network-check-target-wklqk" (UID: "5d5f021d-6241-485a-8611-9fd0beb2ac98") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:48.762051 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:48.762004    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/451de579-98aa-48be-aeee-cf7d09759d3e-original-pull-secret\") pod \"global-pull-secret-syncer-6dj5q\" (UID: \"451de579-98aa-48be-aeee-cf7d09759d3e\") " pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:24:48.762222 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:48.762154    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:48.762222 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:48.762201    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/451de579-98aa-48be-aeee-cf7d09759d3e-original-pull-secret podName:451de579-98aa-48be-aeee-cf7d09759d3e nodeName:}" failed. No retries permitted until 2026-09-16 17:24:56.762188712 +0000 UTC m=+18.124867477 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/451de579-98aa-48be-aeee-cf7d09759d3e-original-pull-secret") pod "global-pull-secret-syncer-6dj5q" (UID: "451de579-98aa-48be-aeee-cf7d09759d3e") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:49.078772 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:49.077529    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:24:49.078772 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:49.077746    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shvlb" podUID="0e392d83-bd3a-4d0b-ac4f-300a6019c434"
Sep 16 17:24:49.078772 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:49.078264    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:24:49.078772 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:49.078375    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dj5q" podUID="451de579-98aa-48be-aeee-cf7d09759d3e"
Sep 16 17:24:50.074889 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:50.074783    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wklqk"
Sep 16 17:24:50.075061 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:50.074924    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wklqk" podUID="5d5f021d-6241-485a-8611-9fd0beb2ac98"
Sep 16 17:24:51.074996 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:51.074960    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:24:51.075457 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:51.074960    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:24:51.075457 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:51.075098    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dj5q" podUID="451de579-98aa-48be-aeee-cf7d09759d3e"
Sep 16 17:24:51.075457 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:51.075162    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shvlb" podUID="0e392d83-bd3a-4d0b-ac4f-300a6019c434"
Sep 16 17:24:52.074706 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:52.074676    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wklqk"
Sep 16 17:24:52.074916 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:52.074775    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wklqk" podUID="5d5f021d-6241-485a-8611-9fd0beb2ac98"
Sep 16 17:24:53.074701 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:53.074611    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:24:53.075143 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:53.074766    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shvlb" podUID="0e392d83-bd3a-4d0b-ac4f-300a6019c434"
Sep 16 17:24:53.075143 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:53.074832    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:24:53.075143 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:53.074937    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dj5q" podUID="451de579-98aa-48be-aeee-cf7d09759d3e"
Sep 16 17:24:54.074167 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:54.074134    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wklqk"
Sep 16 17:24:54.074336 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:54.074266    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wklqk" podUID="5d5f021d-6241-485a-8611-9fd0beb2ac98"
Sep 16 17:24:55.074601 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:55.074393    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:24:55.075052 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:55.074393    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:24:55.075052 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:55.074705    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dj5q" podUID="451de579-98aa-48be-aeee-cf7d09759d3e"
Sep 16 17:24:55.075052 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:55.074822    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shvlb" podUID="0e392d83-bd3a-4d0b-ac4f-300a6019c434"
Sep 16 17:24:56.074652 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:56.074618    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wklqk"
Sep 16 17:24:56.075099 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:56.074723    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wklqk" podUID="5d5f021d-6241-485a-8611-9fd0beb2ac98"
Sep 16 17:24:56.721582 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:56.721544    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd4c6\" (UniqueName: \"kubernetes.io/projected/5d5f021d-6241-485a-8611-9fd0beb2ac98-kube-api-access-jd4c6\") pod \"network-check-target-wklqk\" (UID: \"5d5f021d-6241-485a-8611-9fd0beb2ac98\") " pod="openshift-network-diagnostics/network-check-target-wklqk"
Sep 16 17:24:56.721804 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:56.721630    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e392d83-bd3a-4d0b-ac4f-300a6019c434-metrics-certs\") pod \"network-metrics-daemon-shvlb\" (UID: \"0e392d83-bd3a-4d0b-ac4f-300a6019c434\") " pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:24:56.721804 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:56.721746    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:24:56.721804 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:56.721783    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:24:56.721804 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:56.721797    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jd4c6 for pod openshift-network-diagnostics/network-check-target-wklqk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:56.722045 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:56.721758    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:56.722045 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:56.721854    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d5f021d-6241-485a-8611-9fd0beb2ac98-kube-api-access-jd4c6 podName:5d5f021d-6241-485a-8611-9fd0beb2ac98 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:12.721835233 +0000 UTC m=+34.084514004 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jd4c6" (UniqueName: "kubernetes.io/projected/5d5f021d-6241-485a-8611-9fd0beb2ac98-kube-api-access-jd4c6") pod "network-check-target-wklqk" (UID: "5d5f021d-6241-485a-8611-9fd0beb2ac98") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:56.722045 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:56.721920    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e392d83-bd3a-4d0b-ac4f-300a6019c434-metrics-certs podName:0e392d83-bd3a-4d0b-ac4f-300a6019c434 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:12.721902547 +0000 UTC m=+34.084581317 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e392d83-bd3a-4d0b-ac4f-300a6019c434-metrics-certs") pod "network-metrics-daemon-shvlb" (UID: "0e392d83-bd3a-4d0b-ac4f-300a6019c434") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:56.822392 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:56.822358    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/451de579-98aa-48be-aeee-cf7d09759d3e-original-pull-secret\") pod \"global-pull-secret-syncer-6dj5q\" (UID: \"451de579-98aa-48be-aeee-cf7d09759d3e\") " pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:24:56.822578 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:56.822493    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:56.822578 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:56.822564    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/451de579-98aa-48be-aeee-cf7d09759d3e-original-pull-secret podName:451de579-98aa-48be-aeee-cf7d09759d3e nodeName:}" failed. No retries permitted until 2026-09-16 17:25:12.822541815 +0000 UTC m=+34.185220590 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/451de579-98aa-48be-aeee-cf7d09759d3e-original-pull-secret") pod "global-pull-secret-syncer-6dj5q" (UID: "451de579-98aa-48be-aeee-cf7d09759d3e") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:57.074726 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:57.074630    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:24:57.075183 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:57.074778    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shvlb" podUID="0e392d83-bd3a-4d0b-ac4f-300a6019c434"
Sep 16 17:24:57.075183 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:57.074814    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:24:57.075183 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:57.074916    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dj5q" podUID="451de579-98aa-48be-aeee-cf7d09759d3e"
Sep 16 17:24:58.074879 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:58.074837    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wklqk"
Sep 16 17:24:58.075352 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:58.074956    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wklqk" podUID="5d5f021d-6241-485a-8611-9fd0beb2ac98"
Sep 16 17:24:59.075833 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:59.075795    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:24:59.076299 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:59.075923    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shvlb" podUID="0e392d83-bd3a-4d0b-ac4f-300a6019c434"
Sep 16 17:24:59.076299 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:24:59.075983    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:24:59.076299 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:24:59.076098    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dj5q" podUID="451de579-98aa-48be-aeee-cf7d09759d3e"
Sep 16 17:25:00.074268 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:00.074244    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wklqk"
Sep 16 17:25:00.074359 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:00.074343    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wklqk" podUID="5d5f021d-6241-485a-8611-9fd0beb2ac98"
Sep 16 17:25:01.074430 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:01.074215    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:25:01.075292 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:01.074281    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:25:01.075292 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:01.074528    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dj5q" podUID="451de579-98aa-48be-aeee-cf7d09759d3e"
Sep 16 17:25:01.075292 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:01.074929    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shvlb" podUID="0e392d83-bd3a-4d0b-ac4f-300a6019c434"
Sep 16 17:25:01.317486 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:01.317451    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7rnft" event={"ID":"b674965e-c124-4b4c-ae07-0e17f355f0e0","Type":"ContainerStarted","Data":"9f2fb416ed69ef4669f764d5f69b5133075c8fc58e052463d739206143cf3c2b"}
Sep 16 17:25:01.318954 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:01.318933    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jdgc7" event={"ID":"a202ee1a-5714-4706-9464-7076448f3bf0","Type":"ContainerStarted","Data":"3e7685507c8c5a1e7987fa4d9a51b74cdc2baac11b0a1306810126b294c7711e"}
Sep 16 17:25:01.320277 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:01.320251    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jfsjw" event={"ID":"c6e2238e-ef6c-463e-8123-4663b5961709","Type":"ContainerStarted","Data":"822ab2c4063474e3b47bb635b8da7a280f372c6aa3efd42f4daa85c4616836ef"}
Sep 16 17:25:01.321581 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:01.321554    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9k478" event={"ID":"c910d08c-80cd-486a-98b3-518d769d149f","Type":"ContainerStarted","Data":"738bbbfc0ad73371314bf81830413225d7759b43fca0a969800a57ba685a9a7c"}
Sep 16 17:25:01.322999 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:01.322973    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms5pm" event={"ID":"4c1797a6-6455-460e-a992-8bcd6171acff","Type":"ContainerStarted","Data":"3591f7657d6e5516a2c7ce12db6ba92006bc395c5c27826b4cea8e0cc6e7a4e8"}
Sep 16 17:25:01.324281 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:01.324261    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dwmgh" event={"ID":"ffd13091-dadb-4b9f-bd59-53f1073acedf","Type":"ContainerStarted","Data":"e73246572b54057bcdb558df84206760304f76b84e791bd80b72dcb2be0767dd"}
Sep 16 17:25:01.325707 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:01.325664    2577 generic.go:356] "Generic (PLEG): container finished" podID="74d1b87b-73ed-4d75-b943-e2f8a13808be" containerID="72c77d68c249a1878a3835caf359a86901d3c60f180a792961ca730fff126ad5" exitCode=0
Sep 16 17:25:01.325778 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:01.325725    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njrdx" event={"ID":"74d1b87b-73ed-4d75-b943-e2f8a13808be","Type":"ContainerDied","Data":"72c77d68c249a1878a3835caf359a86901d3c60f180a792961ca730fff126ad5"}
Sep 16 17:25:01.328965 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:01.328947    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5btzs" event={"ID":"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0","Type":"ContainerStarted","Data":"3d529094ac3f719c6a0c17fa549bde937f72418d141d84d4d1ba6ba4b8b53bbd"}
Sep 16 17:25:01.329049 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:01.328997    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5btzs" event={"ID":"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0","Type":"ContainerStarted","Data":"6a4f7cf36c55b3edb3769ce51f392f15858ffdf837d22fbeacbdba96cbb4f4e7"}
Sep 16 17:25:01.329049 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:01.329038    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5btzs" event={"ID":"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0","Type":"ContainerStarted","Data":"00c4aa389cd7670311f2a6c224c463bd59e32a8fdd26d06194869ca32f66266a"}
Sep 16 17:25:01.329124 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:01.329070    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5btzs" event={"ID":"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0","Type":"ContainerStarted","Data":"fc6e71e5de498fcdc5ed5a3c49839bc7680a742cb371f069bbe30cf336e91849"}
Sep 16 17:25:01.329124 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:01.329102    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5btzs" event={"ID":"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0","Type":"ContainerStarted","Data":"6c779f0b8739335ab8f0f87caf0347e696c236d355d2ad29563c0657b0fd6423"}
Sep 16 17:25:01.329183 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:01.329135    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5btzs" event={"ID":"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0","Type":"ContainerStarted","Data":"9a2bbb46b0a023a4468cb1dfec645583b0ca9f0fbe67a9b8f995d2442cfbe610"}
Sep 16 17:25:01.353495 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:01.353461    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7rnft" podStartSLOduration=3.987506933 podStartE2EDuration="22.353449868s" totalImagesPullingTime="18.365942935s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:01.352154702 +0000 UTC m=+22.714833510" watchObservedRunningTime="2026-09-16 17:25:01.353449868 +0000 UTC m=+22.716128654"
Sep 16 17:25:01.420335 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:01.420288    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jfsjw" podStartSLOduration=4.315200778 podStartE2EDuration="22.420274637s" totalImagesPullingTime="18.105073859s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:01.381981975 +0000 UTC m=+22.744660753" watchObservedRunningTime="2026-09-16 17:25:01.420274637 +0000 UTC m=+22.782953405"
Sep 16 17:25:01.458458 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:01.458417    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9k478" podStartSLOduration=11.856349301 podStartE2EDuration="22.458403465s" totalImagesPullingTime="10.602054164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:01.421492949 +0000 UTC m=+22.784171736" watchObservedRunningTime="2026-09-16 17:25:01.458403465 +0000 UTC m=+22.821082298"
Sep 16 17:25:01.458656 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:01.458624    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jdgc7" podStartSLOduration=4.354691508 podStartE2EDuration="22.458615559s" totalImagesPullingTime="18.103924051s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:01.45851296 +0000 UTC m=+22.821191747" watchObservedRunningTime="2026-09-16 17:25:01.458615559 +0000 UTC m=+22.821294346"
Sep 16 17:25:01.562249 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:01.562208    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dwmgh" podStartSLOduration=4.46166849 podStartE2EDuration="22.562193836s" totalImagesPullingTime="18.100525346s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:01.505473414 +0000 UTC m=+22.868152224" watchObservedRunningTime="2026-09-16 17:25:01.562193836 +0000 UTC m=+22.924872621"
Sep 16 17:25:01.733935 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:01.733909    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 17:25:02.058372 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:02.058174    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T17:25:01.733930408Z","UUID":"9c0b228d-ac6f-4341-be36-40b2487d8652","Handler":null,"Name":"","Endpoint":""}
Sep 16 17:25:02.059758 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:02.059735    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 17:25:02.059940 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:02.059772    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 17:25:02.075050 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:02.075027    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wklqk"
Sep 16 17:25:02.075531 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:02.075146    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wklqk" podUID="5d5f021d-6241-485a-8611-9fd0beb2ac98"
Sep 16 17:25:02.333146 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:02.333058    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j2xzd" event={"ID":"445899b8-af25-4084-a9c8-d917110aea4b","Type":"ContainerStarted","Data":"30396f179f2bbd40d55ded9ff65a56b3aca9c47eafb41c5fc5a5e7c44493cdd5"}
Sep 16 17:25:02.335433 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:02.335397    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms5pm" event={"ID":"4c1797a6-6455-460e-a992-8bcd6171acff","Type":"ContainerStarted","Data":"60fa7ddbd5ecb5c654b8fa1fd5fc551a26c617cf25e6df8fbaa6045ec353e925"}
Sep 16 17:25:02.358496 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:02.358440    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-j2xzd" podStartSLOduration=5.257131524 podStartE2EDuration="23.358420434s" totalImagesPullingTime="18.10128891s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:02.358342152 +0000 UTC m=+23.721020940" watchObservedRunningTime="2026-09-16 17:25:02.358420434 +0000 UTC m=+23.721099223"
Sep 16 17:25:03.080004 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:03.079760    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:25:03.080452 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:03.079783    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:25:03.080452 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:03.080106    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dj5q" podUID="451de579-98aa-48be-aeee-cf7d09759d3e"
Sep 16 17:25:03.080452 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:03.080224    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shvlb" podUID="0e392d83-bd3a-4d0b-ac4f-300a6019c434"
Sep 16 17:25:03.341560 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:03.341525    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5btzs" event={"ID":"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0","Type":"ContainerStarted","Data":"f8618b73249450b4e5b469d388fbd53e23344c5e45122ff382352889dfb63877"}
Sep 16 17:25:03.343733 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:03.343705    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms5pm" event={"ID":"4c1797a6-6455-460e-a992-8bcd6171acff","Type":"ContainerStarted","Data":"03ae0f33e5ed8453fb2be177e3beb165d8f33bfe384fa58165684e44cb3afadf"}
Sep 16 17:25:03.380411 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:03.380366    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms5pm" podStartSLOduration=3.8667565870000002 podStartE2EDuration="24.380347293s" totalImagesPullingTime="20.513590706s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:03.380009595 +0000 UTC m=+24.742688384" watchObservedRunningTime="2026-09-16 17:25:03.380347293 +0000 UTC m=+24.743026080"
Sep 16 17:25:03.904691 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:03.904656    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9k478"
Sep 16 17:25:03.905308 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:03.905284    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9k478"
Sep 16 17:25:04.074099 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:04.074070    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wklqk"
Sep 16 17:25:04.074265 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:04.074176    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wklqk" podUID="5d5f021d-6241-485a-8611-9fd0beb2ac98"
Sep 16 17:25:04.346586 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:04.346498    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9k478"
Sep 16 17:25:04.347044 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:04.346897    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9k478"
Sep 16 17:25:05.075137 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:05.075097    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:25:05.075137 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:05.075104    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:25:05.075386 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:05.075251    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shvlb" podUID="0e392d83-bd3a-4d0b-ac4f-300a6019c434"
Sep 16 17:25:05.075386 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:05.075258    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dj5q" podUID="451de579-98aa-48be-aeee-cf7d09759d3e"
Sep 16 17:25:06.075319 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:06.075087    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wklqk"
Sep 16 17:25:06.075673 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:06.075341    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wklqk" podUID="5d5f021d-6241-485a-8611-9fd0beb2ac98"
Sep 16 17:25:06.354910 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:06.354812    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5btzs" event={"ID":"3909b38e-18eb-4b4a-8ec0-0ccb1d6b81c0","Type":"ContainerStarted","Data":"6937d04cc0cfaaa76320ce3ca10e6a0b9c49717aa893350b97c80aaa11bfb0cc"}
Sep 16 17:25:06.355106 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:06.355089    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:25:06.356641 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:06.356619    2577 generic.go:356] "Generic (PLEG): container finished" podID="74d1b87b-73ed-4d75-b943-e2f8a13808be" containerID="7bfa0ceed885b324e0a348035d58ae7260dafd102ab0b82cf87b518b2333a1e7" exitCode=0
Sep 16 17:25:06.356743 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:06.356702    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njrdx" event={"ID":"74d1b87b-73ed-4d75-b943-e2f8a13808be","Type":"ContainerDied","Data":"7bfa0ceed885b324e0a348035d58ae7260dafd102ab0b82cf87b518b2333a1e7"}
Sep 16 17:25:06.369651 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:06.369624    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:25:06.467503 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:06.467452    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5btzs" podStartSLOduration=9.352153415 podStartE2EDuration="27.467438444s" totalImagesPullingTime="18.115285029s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:06.465956877 +0000 UTC m=+27.828635662" watchObservedRunningTime="2026-09-16 17:25:06.467438444 +0000 UTC m=+27.830117229"
Sep 16 17:25:07.074782 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:07.074754    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:25:07.074998 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:07.074754    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:25:07.074998 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:07.074891    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shvlb" podUID="0e392d83-bd3a-4d0b-ac4f-300a6019c434"
Sep 16 17:25:07.074998 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:07.074975    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dj5q" podUID="451de579-98aa-48be-aeee-cf7d09759d3e"
Sep 16 17:25:07.359476 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:07.359404    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:25:07.360166 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:07.359493    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:25:07.374606 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:07.374586    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:25:07.616939 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:07.616856    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wklqk"]
Sep 16 17:25:07.617081 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:07.617022    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wklqk"
Sep 16 17:25:07.617152 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:07.617127    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wklqk" podUID="5d5f021d-6241-485a-8611-9fd0beb2ac98"
Sep 16 17:25:07.623989 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:07.623959    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6dj5q"]
Sep 16 17:25:07.624119 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:07.624078    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:25:07.624212 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:07.624190    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dj5q" podUID="451de579-98aa-48be-aeee-cf7d09759d3e"
Sep 16 17:25:07.624617 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:07.624586    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-shvlb"]
Sep 16 17:25:07.624797 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:07.624696    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:25:07.624850 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:07.624799    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shvlb" podUID="0e392d83-bd3a-4d0b-ac4f-300a6019c434"
Sep 16 17:25:08.365185 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:08.364980    2577 generic.go:356] "Generic (PLEG): container finished" podID="74d1b87b-73ed-4d75-b943-e2f8a13808be" containerID="36d7856f0352f97952507418d030a444fce883e484295a01cef86b167bc74cf7" exitCode=0
Sep 16 17:25:08.365750 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:08.365056    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njrdx" event={"ID":"74d1b87b-73ed-4d75-b943-e2f8a13808be","Type":"ContainerDied","Data":"36d7856f0352f97952507418d030a444fce883e484295a01cef86b167bc74cf7"}
Sep 16 17:25:09.076314 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:09.076283    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:25:09.076496 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:09.076327    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wklqk"
Sep 16 17:25:09.076496 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:09.076403    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:25:09.076496 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:09.076430    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shvlb" podUID="0e392d83-bd3a-4d0b-ac4f-300a6019c434"
Sep 16 17:25:09.076496 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:09.076481    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dj5q" podUID="451de579-98aa-48be-aeee-cf7d09759d3e"
Sep 16 17:25:09.076700 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:09.076545    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wklqk" podUID="5d5f021d-6241-485a-8611-9fd0beb2ac98"
Sep 16 17:25:10.371438 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:10.371344    2577 generic.go:356] "Generic (PLEG): container finished" podID="74d1b87b-73ed-4d75-b943-e2f8a13808be" containerID="5a0c50e240eee23cefd207d089c14fe7ca31beb711cb00723f3eb1e2cc9eff75" exitCode=0
Sep 16 17:25:10.371438 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:10.371385    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njrdx" event={"ID":"74d1b87b-73ed-4d75-b943-e2f8a13808be","Type":"ContainerDied","Data":"5a0c50e240eee23cefd207d089c14fe7ca31beb711cb00723f3eb1e2cc9eff75"}
Sep 16 17:25:11.074487 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:11.074452    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wklqk"
Sep 16 17:25:11.074675 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:11.074495    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:25:11.074675 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:11.074571    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:25:11.074675 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:11.074586    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wklqk" podUID="5d5f021d-6241-485a-8611-9fd0beb2ac98"
Sep 16 17:25:11.074805 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:11.074673    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shvlb" podUID="0e392d83-bd3a-4d0b-ac4f-300a6019c434"
Sep 16 17:25:11.074805 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:11.074759    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dj5q" podUID="451de579-98aa-48be-aeee-cf7d09759d3e"
Sep 16 17:25:12.742522 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:12.742435    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd4c6\" (UniqueName: \"kubernetes.io/projected/5d5f021d-6241-485a-8611-9fd0beb2ac98-kube-api-access-jd4c6\") pod \"network-check-target-wklqk\" (UID: \"5d5f021d-6241-485a-8611-9fd0beb2ac98\") " pod="openshift-network-diagnostics/network-check-target-wklqk"
Sep 16 17:25:12.742993 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:12.742534    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e392d83-bd3a-4d0b-ac4f-300a6019c434-metrics-certs\") pod \"network-metrics-daemon-shvlb\" (UID: \"0e392d83-bd3a-4d0b-ac4f-300a6019c434\") " pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:25:12.742993 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:12.742624    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:25:12.742993 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:12.742643    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:25:12.742993 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:12.742649    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:25:12.742993 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:12.742662    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jd4c6 for pod openshift-network-diagnostics/network-check-target-wklqk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:25:12.742993 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:12.742709    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e392d83-bd3a-4d0b-ac4f-300a6019c434-metrics-certs podName:0e392d83-bd3a-4d0b-ac4f-300a6019c434 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:44.742690056 +0000 UTC m=+66.105368844 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e392d83-bd3a-4d0b-ac4f-300a6019c434-metrics-certs") pod "network-metrics-daemon-shvlb" (UID: "0e392d83-bd3a-4d0b-ac4f-300a6019c434") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:25:12.742993 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:12.742728    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d5f021d-6241-485a-8611-9fd0beb2ac98-kube-api-access-jd4c6 podName:5d5f021d-6241-485a-8611-9fd0beb2ac98 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:44.742717693 +0000 UTC m=+66.105396461 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jd4c6" (UniqueName: "kubernetes.io/projected/5d5f021d-6241-485a-8611-9fd0beb2ac98-kube-api-access-jd4c6") pod "network-check-target-wklqk" (UID: "5d5f021d-6241-485a-8611-9fd0beb2ac98") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:25:12.833805 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:12.833774    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:12.843756 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:12.843726    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/451de579-98aa-48be-aeee-cf7d09759d3e-original-pull-secret\") pod \"global-pull-secret-syncer-6dj5q\" (UID: \"451de579-98aa-48be-aeee-cf7d09759d3e\") " pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:25:12.843956 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:12.843934    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:25:12.844021 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:12.844013    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/451de579-98aa-48be-aeee-cf7d09759d3e-original-pull-secret podName:451de579-98aa-48be-aeee-cf7d09759d3e nodeName:}" failed. No retries permitted until 2026-09-16 17:25:44.843994651 +0000 UTC m=+66.206673425 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/451de579-98aa-48be-aeee-cf7d09759d3e-original-pull-secret") pod "global-pull-secret-syncer-6dj5q" (UID: "451de579-98aa-48be-aeee-cf7d09759d3e") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:25:13.056914 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:13.056801    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:13.074801 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:13.074774    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:25:13.074801 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:13.074801    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wklqk"
Sep 16 17:25:13.075024 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:13.074772    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:25:13.075024 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:13.074919    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6dj5q" podUID="451de579-98aa-48be-aeee-cf7d09759d3e"
Sep 16 17:25:13.075112 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:13.075063    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-shvlb" podUID="0e392d83-bd3a-4d0b-ac4f-300a6019c434"
Sep 16 17:25:13.075171 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:13.075152    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wklqk" podUID="5d5f021d-6241-485a-8611-9fd0beb2ac98"
Sep 16 17:25:14.384976 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.384711    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njrdx" event={"ID":"74d1b87b-73ed-4d75-b943-e2f8a13808be","Type":"ContainerStarted","Data":"c514cbceef5e9373614df2588d19d94a9287e0ba1572c0f0f819a11d0c6247b3"}
Sep 16 17:25:14.414077 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.414054    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-122.ec2.internal" event="NodeReady"
Sep 16 17:25:14.414218 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.414208    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 17:25:14.529453 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.529373    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dbvzj"]
Sep 16 17:25:14.532276 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.532244    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-n48jk"]
Sep 16 17:25:14.532403 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.532355    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dbvzj"
Sep 16 17:25:14.535164 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.535147    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n48jk"
Sep 16 17:25:14.538395 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.538377    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 17:25:14.538510 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.538478    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 17:25:14.538570 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.538519    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qld9l\""
Sep 16 17:25:14.539061 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.538949    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 17:25:14.539386 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.539365    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 17:25:14.539591 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.539566    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 17:25:14.539901 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.539888    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dv5nc\""
Sep 16 17:25:14.559077 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.559054    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dbvzj"]
Sep 16 17:25:14.559775 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.559751    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n48jk"]
Sep 16 17:25:14.657472 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.657435    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34b1ddc8-ea48-4109-852d-24db0edad32c-metrics-tls\") pod \"dns-default-dbvzj\" (UID: \"34b1ddc8-ea48-4109-852d-24db0edad32c\") " pod="openshift-dns/dns-default-dbvzj"
Sep 16 17:25:14.657663 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.657492    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/34b1ddc8-ea48-4109-852d-24db0edad32c-config-volume\") pod \"dns-default-dbvzj\" (UID: \"34b1ddc8-ea48-4109-852d-24db0edad32c\") " pod="openshift-dns/dns-default-dbvzj"
Sep 16 17:25:14.657663 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.657546    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dljzb\" (UniqueName: \"kubernetes.io/projected/34b1ddc8-ea48-4109-852d-24db0edad32c-kube-api-access-dljzb\") pod \"dns-default-dbvzj\" (UID: \"34b1ddc8-ea48-4109-852d-24db0edad32c\") " pod="openshift-dns/dns-default-dbvzj"
Sep 16 17:25:14.657663 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.657603    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34b1ddc8-ea48-4109-852d-24db0edad32c-tmp-dir\") pod \"dns-default-dbvzj\" (UID: \"34b1ddc8-ea48-4109-852d-24db0edad32c\") " pod="openshift-dns/dns-default-dbvzj"
Sep 16 17:25:14.657663 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.657636    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1702bbd4-a888-49db-9bb3-eb6ba31fc8f9-cert\") pod \"ingress-canary-n48jk\" (UID: \"1702bbd4-a888-49db-9bb3-eb6ba31fc8f9\") " pod="openshift-ingress-canary/ingress-canary-n48jk"
Sep 16 17:25:14.657663 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.657662    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c5pp\" (UniqueName: \"kubernetes.io/projected/1702bbd4-a888-49db-9bb3-eb6ba31fc8f9-kube-api-access-8c5pp\") pod \"ingress-canary-n48jk\" (UID: \"1702bbd4-a888-49db-9bb3-eb6ba31fc8f9\") " pod="openshift-ingress-canary/ingress-canary-n48jk"
Sep 16 17:25:14.758476 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.758442    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34b1ddc8-ea48-4109-852d-24db0edad32c-metrics-tls\") pod \"dns-default-dbvzj\" (UID: \"34b1ddc8-ea48-4109-852d-24db0edad32c\") " pod="openshift-dns/dns-default-dbvzj"
Sep 16 17:25:14.758476 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.758489    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/34b1ddc8-ea48-4109-852d-24db0edad32c-config-volume\") pod \"dns-default-dbvzj\" (UID: \"34b1ddc8-ea48-4109-852d-24db0edad32c\") " pod="openshift-dns/dns-default-dbvzj"
Sep 16 17:25:14.758680 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.758509    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dljzb\" (UniqueName: \"kubernetes.io/projected/34b1ddc8-ea48-4109-852d-24db0edad32c-kube-api-access-dljzb\") pod \"dns-default-dbvzj\" (UID: \"34b1ddc8-ea48-4109-852d-24db0edad32c\") " pod="openshift-dns/dns-default-dbvzj"
Sep 16 17:25:14.758680 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:14.758602    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:14.758680 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.758629    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34b1ddc8-ea48-4109-852d-24db0edad32c-tmp-dir\") pod \"dns-default-dbvzj\" (UID: \"34b1ddc8-ea48-4109-852d-24db0edad32c\") " pod="openshift-dns/dns-default-dbvzj"
Sep 16 17:25:14.758680 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:14.758671    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34b1ddc8-ea48-4109-852d-24db0edad32c-metrics-tls podName:34b1ddc8-ea48-4109-852d-24db0edad32c nodeName:}" failed. No retries permitted until 2026-09-16 17:25:15.258650815 +0000 UTC m=+36.621329583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34b1ddc8-ea48-4109-852d-24db0edad32c-metrics-tls") pod "dns-default-dbvzj" (UID: "34b1ddc8-ea48-4109-852d-24db0edad32c") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:14.758893 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.758687    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1702bbd4-a888-49db-9bb3-eb6ba31fc8f9-cert\") pod \"ingress-canary-n48jk\" (UID: \"1702bbd4-a888-49db-9bb3-eb6ba31fc8f9\") " pod="openshift-ingress-canary/ingress-canary-n48jk"
Sep 16 17:25:14.758893 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.758715    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c5pp\" (UniqueName: \"kubernetes.io/projected/1702bbd4-a888-49db-9bb3-eb6ba31fc8f9-kube-api-access-8c5pp\") pod \"ingress-canary-n48jk\" (UID: \"1702bbd4-a888-49db-9bb3-eb6ba31fc8f9\") " pod="openshift-ingress-canary/ingress-canary-n48jk"
Sep 16 17:25:14.758993 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.758962    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34b1ddc8-ea48-4109-852d-24db0edad32c-tmp-dir\") pod \"dns-default-dbvzj\" (UID: \"34b1ddc8-ea48-4109-852d-24db0edad32c\") " pod="openshift-dns/dns-default-dbvzj"
Sep 16 17:25:14.759162 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.759145    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/34b1ddc8-ea48-4109-852d-24db0edad32c-config-volume\") pod \"dns-default-dbvzj\" (UID: \"34b1ddc8-ea48-4109-852d-24db0edad32c\") " pod="openshift-dns/dns-default-dbvzj"
Sep 16 17:25:14.762698 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.762680    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1702bbd4-a888-49db-9bb3-eb6ba31fc8f9-cert\") pod \"ingress-canary-n48jk\" (UID: \"1702bbd4-a888-49db-9bb3-eb6ba31fc8f9\") " pod="openshift-ingress-canary/ingress-canary-n48jk"
Sep 16 17:25:14.772072 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.772051    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dljzb\" (UniqueName: \"kubernetes.io/projected/34b1ddc8-ea48-4109-852d-24db0edad32c-kube-api-access-dljzb\") pod \"dns-default-dbvzj\" (UID: \"34b1ddc8-ea48-4109-852d-24db0edad32c\") " pod="openshift-dns/dns-default-dbvzj"
Sep 16 17:25:14.774942 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.774918    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c5pp\" (UniqueName: \"kubernetes.io/projected/1702bbd4-a888-49db-9bb3-eb6ba31fc8f9-kube-api-access-8c5pp\") pod \"ingress-canary-n48jk\" (UID: \"1702bbd4-a888-49db-9bb3-eb6ba31fc8f9\") " pod="openshift-ingress-canary/ingress-canary-n48jk"
Sep 16 17:25:14.846966 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:14.846877    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n48jk"
Sep 16 17:25:15.074588 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:15.074554    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:25:15.074744 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:15.074567    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:25:15.074744 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:15.074567    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wklqk"
Sep 16 17:25:15.081735 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:15.081710    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 17:25:15.081903 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:15.081768    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 17:25:15.081903 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:15.081777    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 17:25:15.082085 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:15.082070    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vhcq5\""
Sep 16 17:25:15.082142 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:15.082070    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hp4r4\""
Sep 16 17:25:15.082880 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:15.082842    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 17:25:15.142422 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:15.142344    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n48jk"]
Sep 16 17:25:15.146302 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:25:15.146278    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1702bbd4_a888_49db_9bb3_eb6ba31fc8f9.slice/crio-e8b6b8bbcfc1d7f169358e0b000dfb2efe570ac50721f86d589b02bf430354e2 WatchSource:0}: Error finding container e8b6b8bbcfc1d7f169358e0b000dfb2efe570ac50721f86d589b02bf430354e2: Status 404 returned error can't find the container with id e8b6b8bbcfc1d7f169358e0b000dfb2efe570ac50721f86d589b02bf430354e2
Sep 16 17:25:15.261500 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:15.261469    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34b1ddc8-ea48-4109-852d-24db0edad32c-metrics-tls\") pod \"dns-default-dbvzj\" (UID: \"34b1ddc8-ea48-4109-852d-24db0edad32c\") " pod="openshift-dns/dns-default-dbvzj"
Sep 16 17:25:15.261652 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:15.261627    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:15.261724 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:15.261713    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34b1ddc8-ea48-4109-852d-24db0edad32c-metrics-tls podName:34b1ddc8-ea48-4109-852d-24db0edad32c nodeName:}" failed. No retries permitted until 2026-09-16 17:25:16.261697162 +0000 UTC m=+37.624375939 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34b1ddc8-ea48-4109-852d-24db0edad32c-metrics-tls") pod "dns-default-dbvzj" (UID: "34b1ddc8-ea48-4109-852d-24db0edad32c") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:15.392396 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:15.392365    2577 generic.go:356] "Generic (PLEG): container finished" podID="74d1b87b-73ed-4d75-b943-e2f8a13808be" containerID="c514cbceef5e9373614df2588d19d94a9287e0ba1572c0f0f819a11d0c6247b3" exitCode=0
Sep 16 17:25:15.393276 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:15.392433    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njrdx" event={"ID":"74d1b87b-73ed-4d75-b943-e2f8a13808be","Type":"ContainerDied","Data":"c514cbceef5e9373614df2588d19d94a9287e0ba1572c0f0f819a11d0c6247b3"}
Sep 16 17:25:15.393529 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:15.393508    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n48jk" event={"ID":"1702bbd4-a888-49db-9bb3-eb6ba31fc8f9","Type":"ContainerStarted","Data":"e8b6b8bbcfc1d7f169358e0b000dfb2efe570ac50721f86d589b02bf430354e2"}
Sep 16 17:25:16.269603 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:16.269573    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34b1ddc8-ea48-4109-852d-24db0edad32c-metrics-tls\") pod \"dns-default-dbvzj\" (UID: \"34b1ddc8-ea48-4109-852d-24db0edad32c\") " pod="openshift-dns/dns-default-dbvzj"
Sep 16 17:25:16.269784 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:16.269696    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:16.269784 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:16.269751    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34b1ddc8-ea48-4109-852d-24db0edad32c-metrics-tls podName:34b1ddc8-ea48-4109-852d-24db0edad32c nodeName:}" failed. No retries permitted until 2026-09-16 17:25:18.269738766 +0000 UTC m=+39.632417530 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34b1ddc8-ea48-4109-852d-24db0edad32c-metrics-tls") pod "dns-default-dbvzj" (UID: "34b1ddc8-ea48-4109-852d-24db0edad32c") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:16.400960 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:16.400702    2577 generic.go:356] "Generic (PLEG): container finished" podID="74d1b87b-73ed-4d75-b943-e2f8a13808be" containerID="b370daca21c22a7c51e97cc97ca3e1159620d2a6674767c14b486edbae3552f0" exitCode=0
Sep 16 17:25:16.401368 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:16.400783    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njrdx" event={"ID":"74d1b87b-73ed-4d75-b943-e2f8a13808be","Type":"ContainerDied","Data":"b370daca21c22a7c51e97cc97ca3e1159620d2a6674767c14b486edbae3552f0"}
Sep 16 17:25:16.508282 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:16.508249    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:16.717944 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:16.717820    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:17.409046 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:17.408937    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njrdx" event={"ID":"74d1b87b-73ed-4d75-b943-e2f8a13808be","Type":"ContainerStarted","Data":"3178696d0777c1f473b2da089ca30961c59492e3b11f8f6a058106ed207d2bcd"}
Sep 16 17:25:17.473161 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:17.473114    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-njrdx" podStartSLOduration=10.321421858 podStartE2EDuration="38.473097878s" totalImagesPullingTime="28.15167602s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:17.472910337 +0000 UTC m=+38.835589124" watchObservedRunningTime="2026-09-16 17:25:17.473097878 +0000 UTC m=+38.835776662"
Sep 16 17:25:17.671575 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:17.671486    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:17.811813 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:17.811780    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:18.210314 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:18.210239    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:18.287283 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:18.287242    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34b1ddc8-ea48-4109-852d-24db0edad32c-metrics-tls\") pod \"dns-default-dbvzj\" (UID: \"34b1ddc8-ea48-4109-852d-24db0edad32c\") " pod="openshift-dns/dns-default-dbvzj"
Sep 16 17:25:18.287429 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:18.287397    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:18.287467 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:18.287461    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34b1ddc8-ea48-4109-852d-24db0edad32c-metrics-tls podName:34b1ddc8-ea48-4109-852d-24db0edad32c nodeName:}" failed. No retries permitted until 2026-09-16 17:25:22.287445491 +0000 UTC m=+43.650124271 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34b1ddc8-ea48-4109-852d-24db0edad32c-metrics-tls") pod "dns-default-dbvzj" (UID: "34b1ddc8-ea48-4109-852d-24db0edad32c") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:18.413075 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:18.413033    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n48jk" event={"ID":"1702bbd4-a888-49db-9bb3-eb6ba31fc8f9","Type":"ContainerStarted","Data":"54bfe84aa6621e9650992c3e7ab6a780816bb52dccd064a7f3c7c1d560399950"}
Sep 16 17:25:18.469581 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:18.469512    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:18.498550 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:18.498505    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-n48jk" podStartSLOduration=1.802055945 podStartE2EDuration="4.498488962s" totalImagesPullingTime="2.696433017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:25:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:18.497507231 +0000 UTC m=+39.860186017" watchObservedRunningTime="2026-09-16 17:25:18.498488962 +0000 UTC m=+39.861167749"
Sep 16 17:25:20.547708 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:20.547676    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:20.657255 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:20.657221    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:20.773905 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:20.773854    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:22.312827 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:22.312787    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34b1ddc8-ea48-4109-852d-24db0edad32c-metrics-tls\") pod \"dns-default-dbvzj\" (UID: \"34b1ddc8-ea48-4109-852d-24db0edad32c\") " pod="openshift-dns/dns-default-dbvzj"
Sep 16 17:25:22.313228 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:22.312957    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:22.313228 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:22.313022    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34b1ddc8-ea48-4109-852d-24db0edad32c-metrics-tls podName:34b1ddc8-ea48-4109-852d-24db0edad32c nodeName:}" failed. No retries permitted until 2026-09-16 17:25:30.313007293 +0000 UTC m=+51.675686074 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34b1ddc8-ea48-4109-852d-24db0edad32c-metrics-tls") pod "dns-default-dbvzj" (UID: "34b1ddc8-ea48-4109-852d-24db0edad32c") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:22.486213 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:22.486181    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:22.648140 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:22.648057    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:23.916678 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:23.916638    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:24.098795 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:24.098766    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:30.367536 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:30.367497    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34b1ddc8-ea48-4109-852d-24db0edad32c-metrics-tls\") pod \"dns-default-dbvzj\" (UID: \"34b1ddc8-ea48-4109-852d-24db0edad32c\") " pod="openshift-dns/dns-default-dbvzj"
Sep 16 17:25:30.367934 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:30.367636    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:30.367934 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:30.367701    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34b1ddc8-ea48-4109-852d-24db0edad32c-metrics-tls podName:34b1ddc8-ea48-4109-852d-24db0edad32c nodeName:}" failed. No retries permitted until 2026-09-16 17:25:46.36768649 +0000 UTC m=+67.730365258 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34b1ddc8-ea48-4109-852d-24db0edad32c-metrics-tls") pod "dns-default-dbvzj" (UID: "34b1ddc8-ea48-4109-852d-24db0edad32c") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:37.154135 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:37.154102    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:37.239762 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:37.239731    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:38.679369 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:38.679330    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fb94778b8-d97jb"]
Sep 16 17:25:38.704774 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:38.704751    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fb94778b8-d97jb"]
Sep 16 17:25:38.704931 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:38.704854    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fb94778b8-d97jb"
Sep 16 17:25:38.708426 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:38.708406    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 17:25:38.709534 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:38.709516    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 16 17:25:38.709667 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:38.709554    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 17:25:38.709667 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:38.709571    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 17:25:38.709667 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:38.709554    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-ppkfd\""
Sep 16 17:25:38.825316 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:38.825279    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t7wc\" (UniqueName: \"kubernetes.io/projected/3df2e9f8-8583-4672-bde5-56af14edd107-kube-api-access-8t7wc\") pod \"managed-serviceaccount-addon-agent-5fb94778b8-d97jb\" (UID: \"3df2e9f8-8583-4672-bde5-56af14edd107\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fb94778b8-d97jb"
Sep 16 17:25:38.825484 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:38.825326    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3df2e9f8-8583-4672-bde5-56af14edd107-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5fb94778b8-d97jb\" (UID: \"3df2e9f8-8583-4672-bde5-56af14edd107\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fb94778b8-d97jb"
Sep 16 17:25:38.926092 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:38.926057    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t7wc\" (UniqueName: \"kubernetes.io/projected/3df2e9f8-8583-4672-bde5-56af14edd107-kube-api-access-8t7wc\") pod \"managed-serviceaccount-addon-agent-5fb94778b8-d97jb\" (UID: \"3df2e9f8-8583-4672-bde5-56af14edd107\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fb94778b8-d97jb"
Sep 16 17:25:38.926253 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:38.926100    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3df2e9f8-8583-4672-bde5-56af14edd107-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5fb94778b8-d97jb\" (UID: \"3df2e9f8-8583-4672-bde5-56af14edd107\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fb94778b8-d97jb"
Sep 16 17:25:38.929741 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:38.929687    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3df2e9f8-8583-4672-bde5-56af14edd107-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5fb94778b8-d97jb\" (UID: \"3df2e9f8-8583-4672-bde5-56af14edd107\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fb94778b8-d97jb"
Sep 16 17:25:38.934767 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:38.934744    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t7wc\" (UniqueName: \"kubernetes.io/projected/3df2e9f8-8583-4672-bde5-56af14edd107-kube-api-access-8t7wc\") pod \"managed-serviceaccount-addon-agent-5fb94778b8-d97jb\" (UID: \"3df2e9f8-8583-4672-bde5-56af14edd107\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fb94778b8-d97jb"
Sep 16 17:25:39.028086 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:39.028050    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 17:25:39.038218 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:39.038188    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-ppkfd\""
Sep 16 17:25:39.045712 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:39.045687    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fb94778b8-d97jb"
Sep 16 17:25:39.335306 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:39.335275    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fb94778b8-d97jb"]
Sep 16 17:25:39.339686 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:25:39.339658    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3df2e9f8_8583_4672_bde5_56af14edd107.slice/crio-a208a9fc562c493bbf9710b58c4cbbd1b2e8b2c4097fe2e01270608cb12bc1bf WatchSource:0}: Error finding container a208a9fc562c493bbf9710b58c4cbbd1b2e8b2c4097fe2e01270608cb12bc1bf: Status 404 returned error can't find the container with id a208a9fc562c493bbf9710b58c4cbbd1b2e8b2c4097fe2e01270608cb12bc1bf
Sep 16 17:25:39.378037 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:39.378011    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5btzs"
Sep 16 17:25:39.464221 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:39.464188    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fb94778b8-d97jb" event={"ID":"3df2e9f8-8583-4672-bde5-56af14edd107","Type":"ContainerStarted","Data":"a208a9fc562c493bbf9710b58c4cbbd1b2e8b2c4097fe2e01270608cb12bc1bf"}
Sep 16 17:25:39.548928 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:39.548886    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:39.618743 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:39.618655    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:42.474262 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:42.474228    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fb94778b8-d97jb" event={"ID":"3df2e9f8-8583-4672-bde5-56af14edd107","Type":"ContainerStarted","Data":"3b45d0a8ef84d9d97d0504ea19914a41f1c6c1283c99975604ec8f9913b8b91a"}
Sep 16 17:25:42.494202 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:42.494157    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fb94778b8-d97jb" podStartSLOduration=2.3972866330000002 podStartE2EDuration="4.494141605s" totalImagesPullingTime="2.096854972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:25:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:42.493368808 +0000 UTC m=+63.856047594" watchObservedRunningTime="2026-09-16 17:25:42.494141605 +0000 UTC m=+63.856820391"
Sep 16 17:25:44.550184 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:44.550138    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:44.653406 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:44.653373    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:44.759630 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:44.759595    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e392d83-bd3a-4d0b-ac4f-300a6019c434-metrics-certs\") pod \"network-metrics-daemon-shvlb\" (UID: \"0e392d83-bd3a-4d0b-ac4f-300a6019c434\") " pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:25:44.759801 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:44.759651    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd4c6\" (UniqueName: \"kubernetes.io/projected/5d5f021d-6241-485a-8611-9fd0beb2ac98-kube-api-access-jd4c6\") pod \"network-check-target-wklqk\" (UID: \"5d5f021d-6241-485a-8611-9fd0beb2ac98\") " pod="openshift-network-diagnostics/network-check-target-wklqk"
Sep 16 17:25:44.760633 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:44.760600    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:25:44.762806 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:44.762786    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 17:25:44.763728 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:44.763709    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 17:25:44.770830 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:44.770808    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 17:25:44.770922 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:44.770892    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e392d83-bd3a-4d0b-ac4f-300a6019c434-metrics-certs podName:0e392d83-bd3a-4d0b-ac4f-300a6019c434 nodeName:}" failed. No retries permitted until 2026-09-16 17:26:48.77085464 +0000 UTC m=+130.133533413 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e392d83-bd3a-4d0b-ac4f-300a6019c434-metrics-certs") pod "network-metrics-daemon-shvlb" (UID: "0e392d83-bd3a-4d0b-ac4f-300a6019c434") : secret "metrics-daemon-secret" not found
Sep 16 17:25:44.774295 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:44.774281    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 17:25:44.783995 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:44.783968    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd4c6\" (UniqueName: \"kubernetes.io/projected/5d5f021d-6241-485a-8611-9fd0beb2ac98-kube-api-access-jd4c6\") pod \"network-check-target-wklqk\" (UID: \"5d5f021d-6241-485a-8611-9fd0beb2ac98\") " pod="openshift-network-diagnostics/network-check-target-wklqk"
Sep 16 17:25:44.801513 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:44.801464    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vhcq5\""
Sep 16 17:25:44.807563 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:44.807546    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wklqk"
Sep 16 17:25:44.860409 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:44.860375    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/451de579-98aa-48be-aeee-cf7d09759d3e-original-pull-secret\") pod \"global-pull-secret-syncer-6dj5q\" (UID: \"451de579-98aa-48be-aeee-cf7d09759d3e\") " pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:25:44.863572 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:44.863549    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 17:25:44.873361 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:44.873336    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/451de579-98aa-48be-aeee-cf7d09759d3e-original-pull-secret\") pod \"global-pull-secret-syncer-6dj5q\" (UID: \"451de579-98aa-48be-aeee-cf7d09759d3e\") " pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:25:45.081959 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:45.081796    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wklqk"]
Sep 16 17:25:45.084244 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:25:45.084218    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d5f021d_6241_485a_8611_9fd0beb2ac98.slice/crio-62f1e42e22178a33e1c368d10e449db4113648c4445fbda671a92c11607083dd WatchSource:0}: Error finding container 62f1e42e22178a33e1c368d10e449db4113648c4445fbda671a92c11607083dd: Status 404 returned error can't find the container with id 62f1e42e22178a33e1c368d10e449db4113648c4445fbda671a92c11607083dd
Sep 16 17:25:45.086052 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:45.086031    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6dj5q"
Sep 16 17:25:45.381167 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:45.381091    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6dj5q"]
Sep 16 17:25:45.384171 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:25:45.384146    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod451de579_98aa_48be_aeee_cf7d09759d3e.slice/crio-c648189d61b86d212673307b8eb29eda6295fd5a50b1d2f863f9bddd2319ce8c WatchSource:0}: Error finding container c648189d61b86d212673307b8eb29eda6295fd5a50b1d2f863f9bddd2319ce8c: Status 404 returned error can't find the container with id c648189d61b86d212673307b8eb29eda6295fd5a50b1d2f863f9bddd2319ce8c
Sep 16 17:25:45.483049 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:45.483017    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wklqk" event={"ID":"5d5f021d-6241-485a-8611-9fd0beb2ac98","Type":"ContainerStarted","Data":"62f1e42e22178a33e1c368d10e449db4113648c4445fbda671a92c11607083dd"}
Sep 16 17:25:45.484067 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:45.484044    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6dj5q" event={"ID":"451de579-98aa-48be-aeee-cf7d09759d3e","Type":"ContainerStarted","Data":"c648189d61b86d212673307b8eb29eda6295fd5a50b1d2f863f9bddd2319ce8c"}
Sep 16 17:25:46.372003 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:46.371965    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34b1ddc8-ea48-4109-852d-24db0edad32c-metrics-tls\") pod \"dns-default-dbvzj\" (UID: \"34b1ddc8-ea48-4109-852d-24db0edad32c\") " pod="openshift-dns/dns-default-dbvzj"
Sep 16 17:25:46.372449 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:46.372111    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:46.372449 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:25:46.372172    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34b1ddc8-ea48-4109-852d-24db0edad32c-metrics-tls podName:34b1ddc8-ea48-4109-852d-24db0edad32c nodeName:}" failed. No retries permitted until 2026-09-16 17:26:18.372153254 +0000 UTC m=+99.734832056 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34b1ddc8-ea48-4109-852d-24db0edad32c-metrics-tls") pod "dns-default-dbvzj" (UID: "34b1ddc8-ea48-4109-852d-24db0edad32c") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:50.498648 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:50.498608    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wklqk" event={"ID":"5d5f021d-6241-485a-8611-9fd0beb2ac98","Type":"ContainerStarted","Data":"843de52137435d7a4e5eff6b182ed4358b3c7f9cbbcd312930c605cb5fb635f3"}
Sep 16 17:25:50.499215 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:50.498735    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wklqk"
Sep 16 17:25:50.500156 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:50.500134    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6dj5q" event={"ID":"451de579-98aa-48be-aeee-cf7d09759d3e","Type":"ContainerStarted","Data":"322a9e245b6d34fc37724d4025b718a823c9c501039a316ed88530d9219686c7"}
Sep 16 17:25:50.515205 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:50.515161    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wklqk" podStartSLOduration=67.124608154 podStartE2EDuration="1m11.515146758s" totalImagesPullingTime="4.390538604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:50.514708338 +0000 UTC m=+71.877387125" watchObservedRunningTime="2026-09-16 17:25:50.515146758 +0000 UTC m=+71.877825544"
Sep 16 17:25:50.537406 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:50.537357    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6dj5q" podStartSLOduration=66.441669927 podStartE2EDuration="1m10.537342084s" totalImagesPullingTime="4.095672157s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:50.53718299 +0000 UTC m=+71.899861777" watchObservedRunningTime="2026-09-16 17:25:50.537342084 +0000 UTC m=+71.900020892"
Sep 16 17:25:51.163491 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:25:51.163456    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:26:18.385951 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:26:18.385909    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34b1ddc8-ea48-4109-852d-24db0edad32c-metrics-tls\") pod \"dns-default-dbvzj\" (UID: \"34b1ddc8-ea48-4109-852d-24db0edad32c\") " pod="openshift-dns/dns-default-dbvzj"
Sep 16 17:26:18.386315 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:26:18.386069    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:26:18.386315 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:26:18.386142    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34b1ddc8-ea48-4109-852d-24db0edad32c-metrics-tls podName:34b1ddc8-ea48-4109-852d-24db0edad32c nodeName:}" failed. No retries permitted until 2026-09-16 17:27:22.386123603 +0000 UTC m=+163.748802370 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34b1ddc8-ea48-4109-852d-24db0edad32c-metrics-tls") pod "dns-default-dbvzj" (UID: "34b1ddc8-ea48-4109-852d-24db0edad32c") : secret "dns-default-metrics-tls" not found
Sep 16 17:26:21.505320 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:26:21.505291    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wklqk"
Sep 16 17:26:46.354824 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:26:46.354786    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:26:48.786178 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:26:48.786135    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e392d83-bd3a-4d0b-ac4f-300a6019c434-metrics-certs\") pod \"network-metrics-daemon-shvlb\" (UID: \"0e392d83-bd3a-4d0b-ac4f-300a6019c434\") " pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:26:48.786577 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:26:48.786293    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 17:26:48.786577 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:26:48.786364    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e392d83-bd3a-4d0b-ac4f-300a6019c434-metrics-certs podName:0e392d83-bd3a-4d0b-ac4f-300a6019c434 nodeName:}" failed. No retries permitted until 2026-09-16 17:28:50.786347564 +0000 UTC m=+252.149026332 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e392d83-bd3a-4d0b-ac4f-300a6019c434-metrics-certs") pod "network-metrics-daemon-shvlb" (UID: "0e392d83-bd3a-4d0b-ac4f-300a6019c434") : secret "metrics-daemon-secret" not found
Sep 16 17:27:02.795357 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.795320    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-8584f4d7d-4fmb9"]
Sep 16 17:27:02.798225 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.798208    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8584f4d7d-4fmb9"
Sep 16 17:27:02.801775 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.801746    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-snms5\""
Sep 16 17:27:02.802021 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.802007    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 16 17:27:02.802270 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.802255    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 16 17:27:02.802350 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.802260    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 16 17:27:02.802648 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.802631    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 17:27:02.802764 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.802746    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 17:27:02.803045 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.803020    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 16 17:27:02.812403 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.812378    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8584f4d7d-4fmb9"]
Sep 16 17:27:02.883568 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.883512    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/448e3488-54f3-43e2-800e-de6b74a9b4f8-service-ca-bundle\") pod \"router-default-8584f4d7d-4fmb9\" (UID: \"448e3488-54f3-43e2-800e-de6b74a9b4f8\") " pod="openshift-ingress/router-default-8584f4d7d-4fmb9"
Sep 16 17:27:02.883568 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.883569    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/448e3488-54f3-43e2-800e-de6b74a9b4f8-metrics-certs\") pod \"router-default-8584f4d7d-4fmb9\" (UID: \"448e3488-54f3-43e2-800e-de6b74a9b4f8\") " pod="openshift-ingress/router-default-8584f4d7d-4fmb9"
Sep 16 17:27:02.883785 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.883602    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrnrs\" (UniqueName: \"kubernetes.io/projected/448e3488-54f3-43e2-800e-de6b74a9b4f8-kube-api-access-nrnrs\") pod \"router-default-8584f4d7d-4fmb9\" (UID: \"448e3488-54f3-43e2-800e-de6b74a9b4f8\") " pod="openshift-ingress/router-default-8584f4d7d-4fmb9"
Sep 16 17:27:02.883785 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.883630    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/448e3488-54f3-43e2-800e-de6b74a9b4f8-default-certificate\") pod \"router-default-8584f4d7d-4fmb9\" (UID: \"448e3488-54f3-43e2-800e-de6b74a9b4f8\") " pod="openshift-ingress/router-default-8584f4d7d-4fmb9"
Sep 16 17:27:02.883785 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.883647    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/448e3488-54f3-43e2-800e-de6b74a9b4f8-stats-auth\") pod \"router-default-8584f4d7d-4fmb9\" (UID: \"448e3488-54f3-43e2-800e-de6b74a9b4f8\") " pod="openshift-ingress/router-default-8584f4d7d-4fmb9"
Sep 16 17:27:02.905143 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.905109    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-6g6v2"]
Sep 16 17:27:02.908132 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.908110    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-6g6v2"
Sep 16 17:27:02.911929 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.911910    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vh29s\""
Sep 16 17:27:02.912233 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.912216    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 16 17:27:02.912395 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.912378    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 17:27:02.912434 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.912407    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 17:27:02.916662 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.916644    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 16 17:27:02.918310 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.918293    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 16 17:27:02.923033 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.923011    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-6g6v2"]
Sep 16 17:27:02.975236 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.975197    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:02.984090 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.984066    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/448e3488-54f3-43e2-800e-de6b74a9b4f8-service-ca-bundle\") pod \"router-default-8584f4d7d-4fmb9\" (UID: \"448e3488-54f3-43e2-800e-de6b74a9b4f8\") " pod="openshift-ingress/router-default-8584f4d7d-4fmb9"
Sep 16 17:27:02.984171 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.984097    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/448e3488-54f3-43e2-800e-de6b74a9b4f8-metrics-certs\") pod \"router-default-8584f4d7d-4fmb9\" (UID: \"448e3488-54f3-43e2-800e-de6b74a9b4f8\") " pod="openshift-ingress/router-default-8584f4d7d-4fmb9"
Sep 16 17:27:02.984171 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.984115    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrnrs\" (UniqueName: \"kubernetes.io/projected/448e3488-54f3-43e2-800e-de6b74a9b4f8-kube-api-access-nrnrs\") pod \"router-default-8584f4d7d-4fmb9\" (UID: \"448e3488-54f3-43e2-800e-de6b74a9b4f8\") " pod="openshift-ingress/router-default-8584f4d7d-4fmb9"
Sep 16 17:27:02.984171 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.984139    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b23b1744-c013-4acf-967c-bfc9972bde31-trusted-ca-bundle\") pod \"insights-operator-5844b856db-6g6v2\" (UID: \"b23b1744-c013-4acf-967c-bfc9972bde31\") " pod="openshift-insights/insights-operator-5844b856db-6g6v2"
Sep 16 17:27:02.984171 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.984155    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b23b1744-c013-4acf-967c-bfc9972bde31-serving-cert\") pod \"insights-operator-5844b856db-6g6v2\" (UID: \"b23b1744-c013-4acf-967c-bfc9972bde31\") " pod="openshift-insights/insights-operator-5844b856db-6g6v2"
Sep 16 17:27:02.984322 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.984189    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/448e3488-54f3-43e2-800e-de6b74a9b4f8-default-certificate\") pod \"router-default-8584f4d7d-4fmb9\" (UID: \"448e3488-54f3-43e2-800e-de6b74a9b4f8\") " pod="openshift-ingress/router-default-8584f4d7d-4fmb9"
Sep 16 17:27:02.984322 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.984209    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/448e3488-54f3-43e2-800e-de6b74a9b4f8-stats-auth\") pod \"router-default-8584f4d7d-4fmb9\" (UID: \"448e3488-54f3-43e2-800e-de6b74a9b4f8\") " pod="openshift-ingress/router-default-8584f4d7d-4fmb9"
Sep 16 17:27:02.984322 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:02.984219    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 17:27:02.984322 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.984223    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrrj2\" (UniqueName: \"kubernetes.io/projected/b23b1744-c013-4acf-967c-bfc9972bde31-kube-api-access-zrrj2\") pod \"insights-operator-5844b856db-6g6v2\" (UID: \"b23b1744-c013-4acf-967c-bfc9972bde31\") " pod="openshift-insights/insights-operator-5844b856db-6g6v2"
Sep 16 17:27:02.984322 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:02.984272    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/448e3488-54f3-43e2-800e-de6b74a9b4f8-service-ca-bundle podName:448e3488-54f3-43e2-800e-de6b74a9b4f8 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:03.484249367 +0000 UTC m=+144.846928153 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/448e3488-54f3-43e2-800e-de6b74a9b4f8-service-ca-bundle") pod "router-default-8584f4d7d-4fmb9" (UID: "448e3488-54f3-43e2-800e-de6b74a9b4f8") : configmap references non-existent config key: service-ca.crt
Sep 16 17:27:02.984322 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:02.984303    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/448e3488-54f3-43e2-800e-de6b74a9b4f8-metrics-certs podName:448e3488-54f3-43e2-800e-de6b74a9b4f8 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:03.484292701 +0000 UTC m=+144.846971469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/448e3488-54f3-43e2-800e-de6b74a9b4f8-metrics-certs") pod "router-default-8584f4d7d-4fmb9" (UID: "448e3488-54f3-43e2-800e-de6b74a9b4f8") : secret "router-metrics-certs-default" not found
Sep 16 17:27:02.984540 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.984349    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b23b1744-c013-4acf-967c-bfc9972bde31-snapshots\") pod \"insights-operator-5844b856db-6g6v2\" (UID: \"b23b1744-c013-4acf-967c-bfc9972bde31\") " pod="openshift-insights/insights-operator-5844b856db-6g6v2"
Sep 16 17:27:02.984540 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.984389    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b23b1744-c013-4acf-967c-bfc9972bde31-tmp\") pod \"insights-operator-5844b856db-6g6v2\" (UID: \"b23b1744-c013-4acf-967c-bfc9972bde31\") " pod="openshift-insights/insights-operator-5844b856db-6g6v2"
Sep 16 17:27:02.984540 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.984437    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b23b1744-c013-4acf-967c-bfc9972bde31-service-ca-bundle\") pod \"insights-operator-5844b856db-6g6v2\" (UID: \"b23b1744-c013-4acf-967c-bfc9972bde31\") " pod="openshift-insights/insights-operator-5844b856db-6g6v2"
Sep 16 17:27:02.986711 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.986683    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/448e3488-54f3-43e2-800e-de6b74a9b4f8-default-certificate\") pod \"router-default-8584f4d7d-4fmb9\" (UID: \"448e3488-54f3-43e2-800e-de6b74a9b4f8\") " pod="openshift-ingress/router-default-8584f4d7d-4fmb9"
Sep 16 17:27:02.986805 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:02.986688    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/448e3488-54f3-43e2-800e-de6b74a9b4f8-stats-auth\") pod \"router-default-8584f4d7d-4fmb9\" (UID: \"448e3488-54f3-43e2-800e-de6b74a9b4f8\") " pod="openshift-ingress/router-default-8584f4d7d-4fmb9"
Sep 16 17:27:03.003661 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.003635    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrnrs\" (UniqueName: \"kubernetes.io/projected/448e3488-54f3-43e2-800e-de6b74a9b4f8-kube-api-access-nrnrs\") pod \"router-default-8584f4d7d-4fmb9\" (UID: \"448e3488-54f3-43e2-800e-de6b74a9b4f8\") " pod="openshift-ingress/router-default-8584f4d7d-4fmb9"
Sep 16 17:27:03.068028 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.067998    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7d66484d65-bqvc5"]
Sep 16 17:27:03.070944 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.070928    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:03.076882 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.076842    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 17:27:03.077026 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.076910    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jnlm5\""
Sep 16 17:27:03.077145 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.077069    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 17:27:03.077568 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.077539    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 17:27:03.084331 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.084311    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 17:27:03.084889 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.084854    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b23b1744-c013-4acf-967c-bfc9972bde31-trusted-ca-bundle\") pod \"insights-operator-5844b856db-6g6v2\" (UID: \"b23b1744-c013-4acf-967c-bfc9972bde31\") " pod="openshift-insights/insights-operator-5844b856db-6g6v2"
Sep 16 17:27:03.084988 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.084974    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b23b1744-c013-4acf-967c-bfc9972bde31-serving-cert\") pod \"insights-operator-5844b856db-6g6v2\" (UID: \"b23b1744-c013-4acf-967c-bfc9972bde31\") " pod="openshift-insights/insights-operator-5844b856db-6g6v2"
Sep 16 17:27:03.085084 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.085009    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrrj2\" (UniqueName: \"kubernetes.io/projected/b23b1744-c013-4acf-967c-bfc9972bde31-kube-api-access-zrrj2\") pod \"insights-operator-5844b856db-6g6v2\" (UID: \"b23b1744-c013-4acf-967c-bfc9972bde31\") " pod="openshift-insights/insights-operator-5844b856db-6g6v2"
Sep 16 17:27:03.085084 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.085033    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b23b1744-c013-4acf-967c-bfc9972bde31-snapshots\") pod \"insights-operator-5844b856db-6g6v2\" (UID: \"b23b1744-c013-4acf-967c-bfc9972bde31\") " pod="openshift-insights/insights-operator-5844b856db-6g6v2"
Sep 16 17:27:03.085084 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.085060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b23b1744-c013-4acf-967c-bfc9972bde31-tmp\") pod \"insights-operator-5844b856db-6g6v2\" (UID: \"b23b1744-c013-4acf-967c-bfc9972bde31\") " pod="openshift-insights/insights-operator-5844b856db-6g6v2"
Sep 16 17:27:03.085224 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.085088    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b23b1744-c013-4acf-967c-bfc9972bde31-service-ca-bundle\") pod \"insights-operator-5844b856db-6g6v2\" (UID: \"b23b1744-c013-4acf-967c-bfc9972bde31\") " pod="openshift-insights/insights-operator-5844b856db-6g6v2"
Sep 16 17:27:03.085705 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.085683    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b23b1744-c013-4acf-967c-bfc9972bde31-snapshots\") pod \"insights-operator-5844b856db-6g6v2\" (UID: \"b23b1744-c013-4acf-967c-bfc9972bde31\") " pod="openshift-insights/insights-operator-5844b856db-6g6v2"
Sep 16 17:27:03.085824 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.085802    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b23b1744-c013-4acf-967c-bfc9972bde31-tmp\") pod \"insights-operator-5844b856db-6g6v2\" (UID: \"b23b1744-c013-4acf-967c-bfc9972bde31\") " pod="openshift-insights/insights-operator-5844b856db-6g6v2"
Sep 16 17:27:03.085902 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.085833    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b23b1744-c013-4acf-967c-bfc9972bde31-trusted-ca-bundle\") pod \"insights-operator-5844b856db-6g6v2\" (UID: \"b23b1744-c013-4acf-967c-bfc9972bde31\") " pod="openshift-insights/insights-operator-5844b856db-6g6v2"
Sep 16 17:27:03.085902 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.085809    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b23b1744-c013-4acf-967c-bfc9972bde31-service-ca-bundle\") pod \"insights-operator-5844b856db-6g6v2\" (UID: \"b23b1744-c013-4acf-967c-bfc9972bde31\") " pod="openshift-insights/insights-operator-5844b856db-6g6v2"
Sep 16 17:27:03.087309 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.087289    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b23b1744-c013-4acf-967c-bfc9972bde31-serving-cert\") pod \"insights-operator-5844b856db-6g6v2\" (UID: \"b23b1744-c013-4acf-967c-bfc9972bde31\") " pod="openshift-insights/insights-operator-5844b856db-6g6v2"
Sep 16 17:27:03.092383 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.092315    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d66484d65-bqvc5"]
Sep 16 17:27:03.097314 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.097296    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrrj2\" (UniqueName: \"kubernetes.io/projected/b23b1744-c013-4acf-967c-bfc9972bde31-kube-api-access-zrrj2\") pod \"insights-operator-5844b856db-6g6v2\" (UID: \"b23b1744-c013-4acf-967c-bfc9972bde31\") " pod="openshift-insights/insights-operator-5844b856db-6g6v2"
Sep 16 17:27:03.151551 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.151521    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jm27w"]
Sep 16 17:27:03.154598 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.154578    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jm27w"
Sep 16 17:27:03.162937 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.162920    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 16 17:27:03.163701 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.163685    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 16 17:27:03.163747 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.163720    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-9pmxk\""
Sep 16 17:27:03.169018 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.169001    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 16 17:27:03.169115 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.169045    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jm27w"]
Sep 16 17:27:03.171641 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.171624    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 16 17:27:03.185708 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.185687    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f9d6a0b7-d55d-4919-b951-eb073d89c853-installation-pull-secrets\") pod \"image-registry-7d66484d65-bqvc5\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") " pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:03.185817 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.185714    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpmfs\" (UniqueName: \"kubernetes.io/projected/f9d6a0b7-d55d-4919-b951-eb073d89c853-kube-api-access-fpmfs\") pod \"image-registry-7d66484d65-bqvc5\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") " pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:03.185817 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.185744    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f9d6a0b7-d55d-4919-b951-eb073d89c853-registry-tls\") pod \"image-registry-7d66484d65-bqvc5\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") " pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:03.185919 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.185827    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f9d6a0b7-d55d-4919-b951-eb073d89c853-image-registry-private-configuration\") pod \"image-registry-7d66484d65-bqvc5\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") " pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:03.185919 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.185855    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f9d6a0b7-d55d-4919-b951-eb073d89c853-registry-certificates\") pod \"image-registry-7d66484d65-bqvc5\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") " pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:03.185919 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.185905    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f9d6a0b7-d55d-4919-b951-eb073d89c853-trusted-ca\") pod \"image-registry-7d66484d65-bqvc5\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") " pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:03.186019 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.185942    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f9d6a0b7-d55d-4919-b951-eb073d89c853-bound-sa-token\") pod \"image-registry-7d66484d65-bqvc5\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") " pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:03.186019 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.185963    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f9d6a0b7-d55d-4919-b951-eb073d89c853-ca-trust-extracted\") pod \"image-registry-7d66484d65-bqvc5\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") " pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:03.187528 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.187507    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:03.217162 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.217132    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-6g6v2"
Sep 16 17:27:03.287322 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.287292    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/146f8eb9-491d-4f58-a764-ba70d2197637-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-jm27w\" (UID: \"146f8eb9-491d-4f58-a764-ba70d2197637\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jm27w"
Sep 16 17:27:03.287507 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.287328    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzpmw\" (UniqueName: \"kubernetes.io/projected/146f8eb9-491d-4f58-a764-ba70d2197637-kube-api-access-nzpmw\") pod \"kube-storage-version-migrator-operator-6c8f84f897-jm27w\" (UID: \"146f8eb9-491d-4f58-a764-ba70d2197637\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jm27w"
Sep 16 17:27:03.287507 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.287359    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f9d6a0b7-d55d-4919-b951-eb073d89c853-registry-tls\") pod \"image-registry-7d66484d65-bqvc5\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") " pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:03.287507 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:03.287448    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 17:27:03.287507 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:03.287461    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d66484d65-bqvc5: secret "image-registry-tls" not found
Sep 16 17:27:03.287634 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:03.287516    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9d6a0b7-d55d-4919-b951-eb073d89c853-registry-tls podName:f9d6a0b7-d55d-4919-b951-eb073d89c853 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:03.787501327 +0000 UTC m=+145.150180091 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f9d6a0b7-d55d-4919-b951-eb073d89c853-registry-tls") pod "image-registry-7d66484d65-bqvc5" (UID: "f9d6a0b7-d55d-4919-b951-eb073d89c853") : secret "image-registry-tls" not found
Sep 16 17:27:03.287634 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.287534    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/146f8eb9-491d-4f58-a764-ba70d2197637-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-jm27w\" (UID: \"146f8eb9-491d-4f58-a764-ba70d2197637\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jm27w"
Sep 16 17:27:03.287634 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.287554    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f9d6a0b7-d55d-4919-b951-eb073d89c853-image-registry-private-configuration\") pod \"image-registry-7d66484d65-bqvc5\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") " pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:03.287634 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.287570    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f9d6a0b7-d55d-4919-b951-eb073d89c853-registry-certificates\") pod \"image-registry-7d66484d65-bqvc5\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") " pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:03.287634 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.287599    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f9d6a0b7-d55d-4919-b951-eb073d89c853-trusted-ca\") pod \"image-registry-7d66484d65-bqvc5\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") " pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:03.287634 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.287619    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpmfs\" (UniqueName: \"kubernetes.io/projected/f9d6a0b7-d55d-4919-b951-eb073d89c853-kube-api-access-fpmfs\") pod \"image-registry-7d66484d65-bqvc5\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") " pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:03.287824 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.287636    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f9d6a0b7-d55d-4919-b951-eb073d89c853-bound-sa-token\") pod \"image-registry-7d66484d65-bqvc5\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") " pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:03.287824 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.287653    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f9d6a0b7-d55d-4919-b951-eb073d89c853-ca-trust-extracted\") pod \"image-registry-7d66484d65-bqvc5\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") " pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:03.287824 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.287693    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f9d6a0b7-d55d-4919-b951-eb073d89c853-installation-pull-secrets\") pod \"image-registry-7d66484d65-bqvc5\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") " pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:03.288616 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.288584    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f9d6a0b7-d55d-4919-b951-eb073d89c853-ca-trust-extracted\") pod \"image-registry-7d66484d65-bqvc5\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") " pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:03.288750 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.288632    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f9d6a0b7-d55d-4919-b951-eb073d89c853-registry-certificates\") pod \"image-registry-7d66484d65-bqvc5\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") " pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:03.288857 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.288841    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f9d6a0b7-d55d-4919-b951-eb073d89c853-trusted-ca\") pod \"image-registry-7d66484d65-bqvc5\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") " pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:03.290578 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.290557    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f9d6a0b7-d55d-4919-b951-eb073d89c853-installation-pull-secrets\") pod \"image-registry-7d66484d65-bqvc5\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") " pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:03.290729 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.290711    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f9d6a0b7-d55d-4919-b951-eb073d89c853-image-registry-private-configuration\") pod \"image-registry-7d66484d65-bqvc5\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") " pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:03.296281 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.296257    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f9d6a0b7-d55d-4919-b951-eb073d89c853-bound-sa-token\") pod \"image-registry-7d66484d65-bqvc5\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") " pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:03.297047 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.297025    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpmfs\" (UniqueName: \"kubernetes.io/projected/f9d6a0b7-d55d-4919-b951-eb073d89c853-kube-api-access-fpmfs\") pod \"image-registry-7d66484d65-bqvc5\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") " pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:03.388315 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.388237    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/146f8eb9-491d-4f58-a764-ba70d2197637-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-jm27w\" (UID: \"146f8eb9-491d-4f58-a764-ba70d2197637\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jm27w"
Sep 16 17:27:03.388315 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.388273    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzpmw\" (UniqueName: \"kubernetes.io/projected/146f8eb9-491d-4f58-a764-ba70d2197637-kube-api-access-nzpmw\") pod \"kube-storage-version-migrator-operator-6c8f84f897-jm27w\" (UID: \"146f8eb9-491d-4f58-a764-ba70d2197637\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jm27w"
Sep 16 17:27:03.388455 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.388408    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/146f8eb9-491d-4f58-a764-ba70d2197637-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-jm27w\" (UID: \"146f8eb9-491d-4f58-a764-ba70d2197637\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jm27w"
Sep 16 17:27:03.388960 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.388942    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/146f8eb9-491d-4f58-a764-ba70d2197637-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-jm27w\" (UID: \"146f8eb9-491d-4f58-a764-ba70d2197637\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jm27w"
Sep 16 17:27:03.390408 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.390392    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/146f8eb9-491d-4f58-a764-ba70d2197637-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-jm27w\" (UID: \"146f8eb9-491d-4f58-a764-ba70d2197637\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jm27w"
Sep 16 17:27:03.397101 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.397076    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzpmw\" (UniqueName: \"kubernetes.io/projected/146f8eb9-491d-4f58-a764-ba70d2197637-kube-api-access-nzpmw\") pod \"kube-storage-version-migrator-operator-6c8f84f897-jm27w\" (UID: \"146f8eb9-491d-4f58-a764-ba70d2197637\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jm27w"
Sep 16 17:27:03.463452 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.463420    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jm27w"
Sep 16 17:27:03.488976 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.488951    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-6g6v2"]
Sep 16 17:27:03.489233 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.489208    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/448e3488-54f3-43e2-800e-de6b74a9b4f8-service-ca-bundle\") pod \"router-default-8584f4d7d-4fmb9\" (UID: \"448e3488-54f3-43e2-800e-de6b74a9b4f8\") " pod="openshift-ingress/router-default-8584f4d7d-4fmb9"
Sep 16 17:27:03.489280 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.489263    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/448e3488-54f3-43e2-800e-de6b74a9b4f8-metrics-certs\") pod \"router-default-8584f4d7d-4fmb9\" (UID: \"448e3488-54f3-43e2-800e-de6b74a9b4f8\") " pod="openshift-ingress/router-default-8584f4d7d-4fmb9"
Sep 16 17:27:03.489398 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:03.489376    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 17:27:03.489455 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:03.489397    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/448e3488-54f3-43e2-800e-de6b74a9b4f8-service-ca-bundle podName:448e3488-54f3-43e2-800e-de6b74a9b4f8 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:04.489374532 +0000 UTC m=+145.852053308 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/448e3488-54f3-43e2-800e-de6b74a9b4f8-service-ca-bundle") pod "router-default-8584f4d7d-4fmb9" (UID: "448e3488-54f3-43e2-800e-de6b74a9b4f8") : configmap references non-existent config key: service-ca.crt
Sep 16 17:27:03.489455 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:03.489447    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/448e3488-54f3-43e2-800e-de6b74a9b4f8-metrics-certs podName:448e3488-54f3-43e2-800e-de6b74a9b4f8 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:04.489429145 +0000 UTC m=+145.852107923 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/448e3488-54f3-43e2-800e-de6b74a9b4f8-metrics-certs") pod "router-default-8584f4d7d-4fmb9" (UID: "448e3488-54f3-43e2-800e-de6b74a9b4f8") : secret "router-metrics-certs-default" not found
Sep 16 17:27:03.493578 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:27:03.493485    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb23b1744_c013_4acf_967c_bfc9972bde31.slice/crio-f96645abe15c8ce3ab33b82f40643802a11d951e200fe3459f0cfaecf0361e69 WatchSource:0}: Error finding container f96645abe15c8ce3ab33b82f40643802a11d951e200fe3459f0cfaecf0361e69: Status 404 returned error can't find the container with id f96645abe15c8ce3ab33b82f40643802a11d951e200fe3459f0cfaecf0361e69
Sep 16 17:27:03.680674 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.680589    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-6g6v2" event={"ID":"b23b1744-c013-4acf-967c-bfc9972bde31","Type":"ContainerStarted","Data":"f96645abe15c8ce3ab33b82f40643802a11d951e200fe3459f0cfaecf0361e69"}
Sep 16 17:27:03.750576 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:27:03.750548    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146f8eb9_491d_4f58_a764_ba70d2197637.slice/crio-f3dc72a1810eabbc6771532b20d929ca36623b086f2db34aced7e1be0f4b41cd WatchSource:0}: Error finding container f3dc72a1810eabbc6771532b20d929ca36623b086f2db34aced7e1be0f4b41cd: Status 404 returned error can't find the container with id f3dc72a1810eabbc6771532b20d929ca36623b086f2db34aced7e1be0f4b41cd
Sep 16 17:27:03.754949 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.754923    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jm27w"]
Sep 16 17:27:03.791786 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:03.791752    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f9d6a0b7-d55d-4919-b951-eb073d89c853-registry-tls\") pod \"image-registry-7d66484d65-bqvc5\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") " pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:03.791946 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:03.791909    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 17:27:03.791946 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:03.791926    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d66484d65-bqvc5: secret "image-registry-tls" not found
Sep 16 17:27:03.792021 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:03.791975    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9d6a0b7-d55d-4919-b951-eb073d89c853-registry-tls podName:f9d6a0b7-d55d-4919-b951-eb073d89c853 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:04.791960837 +0000 UTC m=+146.154639605 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f9d6a0b7-d55d-4919-b951-eb073d89c853-registry-tls") pod "image-registry-7d66484d65-bqvc5" (UID: "f9d6a0b7-d55d-4919-b951-eb073d89c853") : secret "image-registry-tls" not found
Sep 16 17:27:04.502984 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:04.502936    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/448e3488-54f3-43e2-800e-de6b74a9b4f8-service-ca-bundle\") pod \"router-default-8584f4d7d-4fmb9\" (UID: \"448e3488-54f3-43e2-800e-de6b74a9b4f8\") " pod="openshift-ingress/router-default-8584f4d7d-4fmb9"
Sep 16 17:27:04.503401 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:04.503005    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/448e3488-54f3-43e2-800e-de6b74a9b4f8-metrics-certs\") pod \"router-default-8584f4d7d-4fmb9\" (UID: \"448e3488-54f3-43e2-800e-de6b74a9b4f8\") " pod="openshift-ingress/router-default-8584f4d7d-4fmb9"
Sep 16 17:27:04.503401 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:04.503133    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/448e3488-54f3-43e2-800e-de6b74a9b4f8-service-ca-bundle podName:448e3488-54f3-43e2-800e-de6b74a9b4f8 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:06.503110025 +0000 UTC m=+147.865788807 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/448e3488-54f3-43e2-800e-de6b74a9b4f8-service-ca-bundle") pod "router-default-8584f4d7d-4fmb9" (UID: "448e3488-54f3-43e2-800e-de6b74a9b4f8") : configmap references non-existent config key: service-ca.crt
Sep 16 17:27:04.503401 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:04.503159    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 17:27:04.503401 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:04.503212    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/448e3488-54f3-43e2-800e-de6b74a9b4f8-metrics-certs podName:448e3488-54f3-43e2-800e-de6b74a9b4f8 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:06.503196585 +0000 UTC m=+147.865875352 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/448e3488-54f3-43e2-800e-de6b74a9b4f8-metrics-certs") pod "router-default-8584f4d7d-4fmb9" (UID: "448e3488-54f3-43e2-800e-de6b74a9b4f8") : secret "router-metrics-certs-default" not found
Sep 16 17:27:04.684810 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:04.684774    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jm27w" event={"ID":"146f8eb9-491d-4f58-a764-ba70d2197637","Type":"ContainerStarted","Data":"f3dc72a1810eabbc6771532b20d929ca36623b086f2db34aced7e1be0f4b41cd"}
Sep 16 17:27:04.806571 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:04.806475    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f9d6a0b7-d55d-4919-b951-eb073d89c853-registry-tls\") pod \"image-registry-7d66484d65-bqvc5\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") " pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:04.806725 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:04.806651    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 17:27:04.806725 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:04.806671    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d66484d65-bqvc5: secret "image-registry-tls" not found
Sep 16 17:27:04.806828 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:04.806732    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9d6a0b7-d55d-4919-b951-eb073d89c853-registry-tls podName:f9d6a0b7-d55d-4919-b951-eb073d89c853 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:06.806713129 +0000 UTC m=+148.169391902 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f9d6a0b7-d55d-4919-b951-eb073d89c853-registry-tls") pod "image-registry-7d66484d65-bqvc5" (UID: "f9d6a0b7-d55d-4919-b951-eb073d89c853") : secret "image-registry-tls" not found
Sep 16 17:27:04.911839 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:04.911806    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:04.991011 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:04.990951    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:06.519216 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:06.519180    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/448e3488-54f3-43e2-800e-de6b74a9b4f8-service-ca-bundle\") pod \"router-default-8584f4d7d-4fmb9\" (UID: \"448e3488-54f3-43e2-800e-de6b74a9b4f8\") " pod="openshift-ingress/router-default-8584f4d7d-4fmb9"
Sep 16 17:27:06.519569 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:06.519227    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/448e3488-54f3-43e2-800e-de6b74a9b4f8-metrics-certs\") pod \"router-default-8584f4d7d-4fmb9\" (UID: \"448e3488-54f3-43e2-800e-de6b74a9b4f8\") " pod="openshift-ingress/router-default-8584f4d7d-4fmb9"
Sep 16 17:27:06.519569 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:06.519345    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 17:27:06.519569 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:06.519390    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/448e3488-54f3-43e2-800e-de6b74a9b4f8-service-ca-bundle podName:448e3488-54f3-43e2-800e-de6b74a9b4f8 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:10.519364728 +0000 UTC m=+151.882043499 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/448e3488-54f3-43e2-800e-de6b74a9b4f8-service-ca-bundle") pod "router-default-8584f4d7d-4fmb9" (UID: "448e3488-54f3-43e2-800e-de6b74a9b4f8") : configmap references non-existent config key: service-ca.crt
Sep 16 17:27:06.519569 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:06.519420    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/448e3488-54f3-43e2-800e-de6b74a9b4f8-metrics-certs podName:448e3488-54f3-43e2-800e-de6b74a9b4f8 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:10.519409954 +0000 UTC m=+151.882088722 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/448e3488-54f3-43e2-800e-de6b74a9b4f8-metrics-certs") pod "router-default-8584f4d7d-4fmb9" (UID: "448e3488-54f3-43e2-800e-de6b74a9b4f8") : secret "router-metrics-certs-default" not found
Sep 16 17:27:06.692414 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:06.692371    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jm27w" event={"ID":"146f8eb9-491d-4f58-a764-ba70d2197637","Type":"ContainerStarted","Data":"3cd5625bc4788cfd69bbf7c66f183389c78a7bf63d0480aa8b3d44cec5f269e7"}
Sep 16 17:27:06.693743 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:06.693722    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-6g6v2" event={"ID":"b23b1744-c013-4acf-967c-bfc9972bde31","Type":"ContainerStarted","Data":"b26b68c1a7333ff2fc95a743d3aa0cfa642f15106ea1f65d1b7f2c2ce05c05f7"}
Sep 16 17:27:06.767363 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:06.767297    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jm27w" podStartSLOduration=1.815120511 podStartE2EDuration="3.767275233s" totalImagesPullingTime="1.952154722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:06.723010281 +0000 UTC m=+148.085689066" watchObservedRunningTime="2026-09-16 17:27:06.767275233 +0000 UTC m=+148.129954019"
Sep 16 17:27:06.767666 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:06.767631    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-6g6v2" podStartSLOduration=2.561636088 podStartE2EDuration="4.767617203s" totalImagesPullingTime="2.205981115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:06.766599846 +0000 UTC m=+148.129278633" watchObservedRunningTime="2026-09-16 17:27:06.767617203 +0000 UTC m=+148.130295993"
Sep 16 17:27:06.822142 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:06.822107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f9d6a0b7-d55d-4919-b951-eb073d89c853-registry-tls\") pod \"image-registry-7d66484d65-bqvc5\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") " pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:06.822525 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:06.822243    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 17:27:06.822525 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:06.822414    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d66484d65-bqvc5: secret "image-registry-tls" not found
Sep 16 17:27:06.822525 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:06.822497    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9d6a0b7-d55d-4919-b951-eb073d89c853-registry-tls podName:f9d6a0b7-d55d-4919-b951-eb073d89c853 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:10.822473889 +0000 UTC m=+152.185152668 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f9d6a0b7-d55d-4919-b951-eb073d89c853-registry-tls") pod "image-registry-7d66484d65-bqvc5" (UID: "f9d6a0b7-d55d-4919-b951-eb073d89c853") : secret "image-registry-tls" not found
Sep 16 17:27:06.835673 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:06.835649    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:06.996810 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:06.996771    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:07.091297 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:07.091210    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:07.386706 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:07.386634    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:07.426046 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:07.426019    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-49fvb"]
Sep 16 17:27:07.429305 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:07.429286    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-49fvb"
Sep 16 17:27:07.435399 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:07.435184    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 17:27:07.435492 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:07.435474    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hmpn4\""
Sep 16 17:27:07.435829 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:07.435812    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 17:27:07.450075 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:07.450047    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-49fvb"]
Sep 16 17:27:07.528089 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:07.528052    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrnxf\" (UniqueName: \"kubernetes.io/projected/ad27164e-ba4c-429e-a321-67e56945b867-kube-api-access-jrnxf\") pod \"insights-runtime-extractor-49fvb\" (UID: \"ad27164e-ba4c-429e-a321-67e56945b867\") " pod="openshift-insights/insights-runtime-extractor-49fvb"
Sep 16 17:27:07.528481 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:07.528130    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad27164e-ba4c-429e-a321-67e56945b867-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-49fvb\" (UID: \"ad27164e-ba4c-429e-a321-67e56945b867\") " pod="openshift-insights/insights-runtime-extractor-49fvb"
Sep 16 17:27:07.528481 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:07.528190    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad27164e-ba4c-429e-a321-67e56945b867-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-49fvb\" (UID: \"ad27164e-ba4c-429e-a321-67e56945b867\") " pod="openshift-insights/insights-runtime-extractor-49fvb"
Sep 16 17:27:07.528481 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:07.528284    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad27164e-ba4c-429e-a321-67e56945b867-data-volume\") pod \"insights-runtime-extractor-49fvb\" (UID: \"ad27164e-ba4c-429e-a321-67e56945b867\") " pod="openshift-insights/insights-runtime-extractor-49fvb"
Sep 16 17:27:07.528481 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:07.528345    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ad27164e-ba4c-429e-a321-67e56945b867-crio-socket\") pod \"insights-runtime-extractor-49fvb\" (UID: \"ad27164e-ba4c-429e-a321-67e56945b867\") " pod="openshift-insights/insights-runtime-extractor-49fvb"
Sep 16 17:27:07.545182 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:07.545152    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:07.629184 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:07.629148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ad27164e-ba4c-429e-a321-67e56945b867-crio-socket\") pod \"insights-runtime-extractor-49fvb\" (UID: \"ad27164e-ba4c-429e-a321-67e56945b867\") " pod="openshift-insights/insights-runtime-extractor-49fvb"
Sep 16 17:27:07.629315 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:07.629191    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrnxf\" (UniqueName: \"kubernetes.io/projected/ad27164e-ba4c-429e-a321-67e56945b867-kube-api-access-jrnxf\") pod \"insights-runtime-extractor-49fvb\" (UID: \"ad27164e-ba4c-429e-a321-67e56945b867\") " pod="openshift-insights/insights-runtime-extractor-49fvb"
Sep 16 17:27:07.629315 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:07.629250    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad27164e-ba4c-429e-a321-67e56945b867-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-49fvb\" (UID: \"ad27164e-ba4c-429e-a321-67e56945b867\") " pod="openshift-insights/insights-runtime-extractor-49fvb"
Sep 16 17:27:07.629315 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:07.629269    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ad27164e-ba4c-429e-a321-67e56945b867-crio-socket\") pod \"insights-runtime-extractor-49fvb\" (UID: \"ad27164e-ba4c-429e-a321-67e56945b867\") " pod="openshift-insights/insights-runtime-extractor-49fvb"
Sep 16 17:27:07.629486 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:07.629315    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad27164e-ba4c-429e-a321-67e56945b867-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-49fvb\" (UID: \"ad27164e-ba4c-429e-a321-67e56945b867\") " pod="openshift-insights/insights-runtime-extractor-49fvb"
Sep 16 17:27:07.629486 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:07.629369    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad27164e-ba4c-429e-a321-67e56945b867-data-volume\") pod \"insights-runtime-extractor-49fvb\" (UID: \"ad27164e-ba4c-429e-a321-67e56945b867\") " pod="openshift-insights/insights-runtime-extractor-49fvb"
Sep 16 17:27:07.629486 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:07.629373    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:07.629486 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:07.629435    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad27164e-ba4c-429e-a321-67e56945b867-insights-runtime-extractor-tls podName:ad27164e-ba4c-429e-a321-67e56945b867 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:08.129414386 +0000 UTC m=+149.492093154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ad27164e-ba4c-429e-a321-67e56945b867-insights-runtime-extractor-tls") pod "insights-runtime-extractor-49fvb" (UID: "ad27164e-ba4c-429e-a321-67e56945b867") : secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:07.629486 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:07.629465    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 17:27:07.629672 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:07.629525    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad27164e-ba4c-429e-a321-67e56945b867-kube-rbac-proxy-cm podName:ad27164e-ba4c-429e-a321-67e56945b867 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:08.129509285 +0000 UTC m=+149.492188059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ad27164e-ba4c-429e-a321-67e56945b867-kube-rbac-proxy-cm") pod "insights-runtime-extractor-49fvb" (UID: "ad27164e-ba4c-429e-a321-67e56945b867") : configmap "kube-rbac-proxy" not found
Sep 16 17:27:07.629672 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:07.629659    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad27164e-ba4c-429e-a321-67e56945b867-data-volume\") pod \"insights-runtime-extractor-49fvb\" (UID: \"ad27164e-ba4c-429e-a321-67e56945b867\") " pod="openshift-insights/insights-runtime-extractor-49fvb"
Sep 16 17:27:07.648925 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:07.648856    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrnxf\" (UniqueName: \"kubernetes.io/projected/ad27164e-ba4c-429e-a321-67e56945b867-kube-api-access-jrnxf\") pod \"insights-runtime-extractor-49fvb\" (UID: \"ad27164e-ba4c-429e-a321-67e56945b867\") " pod="openshift-insights/insights-runtime-extractor-49fvb"
Sep 16 17:27:07.739912 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:07.739882    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:07.826566 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:07.826533    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:08.019341 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:08.019262    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:08.114224 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:08.114188    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:08.133349 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:08.133322    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad27164e-ba4c-429e-a321-67e56945b867-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-49fvb\" (UID: \"ad27164e-ba4c-429e-a321-67e56945b867\") " pod="openshift-insights/insights-runtime-extractor-49fvb"
Sep 16 17:27:08.133485 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:08.133374    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad27164e-ba4c-429e-a321-67e56945b867-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-49fvb\" (UID: \"ad27164e-ba4c-429e-a321-67e56945b867\") " pod="openshift-insights/insights-runtime-extractor-49fvb"
Sep 16 17:27:08.133485 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:08.133460    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:08.133555 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:08.133482    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 17:27:08.133555 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:08.133529    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad27164e-ba4c-429e-a321-67e56945b867-insights-runtime-extractor-tls podName:ad27164e-ba4c-429e-a321-67e56945b867 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:09.133504019 +0000 UTC m=+150.496182783 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ad27164e-ba4c-429e-a321-67e56945b867-insights-runtime-extractor-tls") pod "insights-runtime-extractor-49fvb" (UID: "ad27164e-ba4c-429e-a321-67e56945b867") : secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:08.133555 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:08.133544    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad27164e-ba4c-429e-a321-67e56945b867-kube-rbac-proxy-cm podName:ad27164e-ba4c-429e-a321-67e56945b867 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:09.13353704 +0000 UTC m=+150.496215804 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ad27164e-ba4c-429e-a321-67e56945b867-kube-rbac-proxy-cm") pod "insights-runtime-extractor-49fvb" (UID: "ad27164e-ba4c-429e-a321-67e56945b867") : configmap "kube-rbac-proxy" not found
Sep 16 17:27:08.920853 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:08.920818    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:09.021176 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:09.021144    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:09.141385 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:09.141350    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad27164e-ba4c-429e-a321-67e56945b867-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-49fvb\" (UID: \"ad27164e-ba4c-429e-a321-67e56945b867\") " pod="openshift-insights/insights-runtime-extractor-49fvb"
Sep 16 17:27:09.141549 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:09.141414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad27164e-ba4c-429e-a321-67e56945b867-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-49fvb\" (UID: \"ad27164e-ba4c-429e-a321-67e56945b867\") " pod="openshift-insights/insights-runtime-extractor-49fvb"
Sep 16 17:27:09.141549 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:09.141482    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 17:27:09.141549 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:09.141497    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:09.141549 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:09.141546    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad27164e-ba4c-429e-a321-67e56945b867-kube-rbac-proxy-cm podName:ad27164e-ba4c-429e-a321-67e56945b867 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:11.141529664 +0000 UTC m=+152.504208428 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ad27164e-ba4c-429e-a321-67e56945b867-kube-rbac-proxy-cm") pod "insights-runtime-extractor-49fvb" (UID: "ad27164e-ba4c-429e-a321-67e56945b867") : configmap "kube-rbac-proxy" not found
Sep 16 17:27:09.141686 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:09.141561    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad27164e-ba4c-429e-a321-67e56945b867-insights-runtime-extractor-tls podName:ad27164e-ba4c-429e-a321-67e56945b867 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:11.141554691 +0000 UTC m=+152.504233455 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ad27164e-ba4c-429e-a321-67e56945b867-insights-runtime-extractor-tls") pod "insights-runtime-extractor-49fvb" (UID: "ad27164e-ba4c-429e-a321-67e56945b867") : secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:09.658742 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:09.658707    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:09.729730 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:09.729697    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:09.917593 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:09.917523    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jfsjw_c6e2238e-ef6c-463e-8123-4663b5961709/dns-node-resolver/0.log"
Sep 16 17:27:10.553692 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:10.553656    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/448e3488-54f3-43e2-800e-de6b74a9b4f8-service-ca-bundle\") pod \"router-default-8584f4d7d-4fmb9\" (UID: \"448e3488-54f3-43e2-800e-de6b74a9b4f8\") " pod="openshift-ingress/router-default-8584f4d7d-4fmb9"
Sep 16 17:27:10.553692 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:10.553697    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/448e3488-54f3-43e2-800e-de6b74a9b4f8-metrics-certs\") pod \"router-default-8584f4d7d-4fmb9\" (UID: \"448e3488-54f3-43e2-800e-de6b74a9b4f8\") " pod="openshift-ingress/router-default-8584f4d7d-4fmb9"
Sep 16 17:27:10.554161 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:10.553812    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 17:27:10.554161 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:10.553829    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/448e3488-54f3-43e2-800e-de6b74a9b4f8-service-ca-bundle podName:448e3488-54f3-43e2-800e-de6b74a9b4f8 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:18.553811564 +0000 UTC m=+159.916490327 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/448e3488-54f3-43e2-800e-de6b74a9b4f8-service-ca-bundle") pod "router-default-8584f4d7d-4fmb9" (UID: "448e3488-54f3-43e2-800e-de6b74a9b4f8") : configmap references non-existent config key: service-ca.crt
Sep 16 17:27:10.554161 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:10.553853    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/448e3488-54f3-43e2-800e-de6b74a9b4f8-metrics-certs podName:448e3488-54f3-43e2-800e-de6b74a9b4f8 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:18.553845832 +0000 UTC m=+159.916524596 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/448e3488-54f3-43e2-800e-de6b74a9b4f8-metrics-certs") pod "router-default-8584f4d7d-4fmb9" (UID: "448e3488-54f3-43e2-800e-de6b74a9b4f8") : secret "router-metrics-certs-default" not found
Sep 16 17:27:10.649652 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:10.649612    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:10.856825 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:10.856789    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f9d6a0b7-d55d-4919-b951-eb073d89c853-registry-tls\") pod \"image-registry-7d66484d65-bqvc5\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") " pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:10.856996 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:10.856957    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 17:27:10.856996 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:10.856977    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d66484d65-bqvc5: secret "image-registry-tls" not found
Sep 16 17:27:10.857068 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:10.857032    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9d6a0b7-d55d-4919-b951-eb073d89c853-registry-tls podName:f9d6a0b7-d55d-4919-b951-eb073d89c853 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:18.857015826 +0000 UTC m=+160.219694595 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f9d6a0b7-d55d-4919-b951-eb073d89c853-registry-tls") pod "image-registry-7d66484d65-bqvc5" (UID: "f9d6a0b7-d55d-4919-b951-eb073d89c853") : secret "image-registry-tls" not found
Sep 16 17:27:10.917134 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:10.917104    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dwmgh_ffd13091-dadb-4b9f-bd59-53f1073acedf/node-ca/0.log"
Sep 16 17:27:11.159265 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:11.159177    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad27164e-ba4c-429e-a321-67e56945b867-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-49fvb\" (UID: \"ad27164e-ba4c-429e-a321-67e56945b867\") " pod="openshift-insights/insights-runtime-extractor-49fvb"
Sep 16 17:27:11.159416 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:11.159272    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad27164e-ba4c-429e-a321-67e56945b867-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-49fvb\" (UID: \"ad27164e-ba4c-429e-a321-67e56945b867\") " pod="openshift-insights/insights-runtime-extractor-49fvb"
Sep 16 17:27:11.159416 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:11.159328    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:11.159416 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:11.159375    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 17:27:11.159416 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:11.159411    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad27164e-ba4c-429e-a321-67e56945b867-insights-runtime-extractor-tls podName:ad27164e-ba4c-429e-a321-67e56945b867 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:15.159388776 +0000 UTC m=+156.522067543 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ad27164e-ba4c-429e-a321-67e56945b867-insights-runtime-extractor-tls") pod "insights-runtime-extractor-49fvb" (UID: "ad27164e-ba4c-429e-a321-67e56945b867") : secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:11.159603 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:11.159432    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad27164e-ba4c-429e-a321-67e56945b867-kube-rbac-proxy-cm podName:ad27164e-ba4c-429e-a321-67e56945b867 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:15.159423145 +0000 UTC m=+156.522101909 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ad27164e-ba4c-429e-a321-67e56945b867-kube-rbac-proxy-cm") pod "insights-runtime-extractor-49fvb" (UID: "ad27164e-ba4c-429e-a321-67e56945b867") : configmap "kube-rbac-proxy" not found
Sep 16 17:27:11.535548 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:11.535474    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:11.720720 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:11.720685    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:11.930995 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:11.930969    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n48jk_1702bbd4-a888-49db-9bb3-eb6ba31fc8f9/serve-healthcheck-canary/0.log"
Sep 16 17:27:12.002809 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:12.002773    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:12.103413 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:12.103381    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:12.222329 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:12.222239    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:12.718946 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:12.718914    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-jm27w_146f8eb9-491d-4f58-a764-ba70d2197637/kube-storage-version-migrator-operator/0.log"
Sep 16 17:27:12.818304 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:12.818269    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:13.444879 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:13.444837    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:14.032532 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:14.032490    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:14.226430 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:14.226389    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:14.295533 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:14.295460    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:14.423566 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:14.423528    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:14.619327 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:14.619268    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:15.195488 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:15.195450    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad27164e-ba4c-429e-a321-67e56945b867-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-49fvb\" (UID: \"ad27164e-ba4c-429e-a321-67e56945b867\") " pod="openshift-insights/insights-runtime-extractor-49fvb"
Sep 16 17:27:15.196021 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:15.195532    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad27164e-ba4c-429e-a321-67e56945b867-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-49fvb\" (UID: \"ad27164e-ba4c-429e-a321-67e56945b867\") " pod="openshift-insights/insights-runtime-extractor-49fvb"
Sep 16 17:27:15.196021 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:15.195590    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 17:27:15.196021 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:15.195621    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:15.196021 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:15.195670    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad27164e-ba4c-429e-a321-67e56945b867-kube-rbac-proxy-cm podName:ad27164e-ba4c-429e-a321-67e56945b867 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:23.195649496 +0000 UTC m=+164.558328550 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ad27164e-ba4c-429e-a321-67e56945b867-kube-rbac-proxy-cm") pod "insights-runtime-extractor-49fvb" (UID: "ad27164e-ba4c-429e-a321-67e56945b867") : configmap "kube-rbac-proxy" not found
Sep 16 17:27:15.196021 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:15.195690    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad27164e-ba4c-429e-a321-67e56945b867-insights-runtime-extractor-tls podName:ad27164e-ba4c-429e-a321-67e56945b867 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:23.195679946 +0000 UTC m=+164.558358710 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ad27164e-ba4c-429e-a321-67e56945b867-insights-runtime-extractor-tls") pod "insights-runtime-extractor-49fvb" (UID: "ad27164e-ba4c-429e-a321-67e56945b867") : secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:15.416449 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:15.416398    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:15.495723 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:15.495636    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:15.611994 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:15.611951    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:15.815804 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:15.815764    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:16.023154 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:16.023124    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:16.124574 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:16.124487    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:16.224252 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:16.224198    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:17.327463 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:17.327432    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:17.461177 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:17.461102    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:17.542749 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:17.542706    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-dbvzj" podUID="34b1ddc8-ea48-4109-852d-24db0edad32c"
Sep 16 17:27:17.727847 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:17.727757    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dbvzj"
Sep 16 17:27:17.830590 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:17.830557    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:18.022482 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:18.022395    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:18.092229 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:18.092189    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-shvlb" podUID="0e392d83-bd3a-4d0b-ac4f-300a6019c434"
Sep 16 17:27:18.126242 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:18.126206    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:18.433119 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:18.433086    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:18.626473 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:18.626436    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/448e3488-54f3-43e2-800e-de6b74a9b4f8-service-ca-bundle\") pod \"router-default-8584f4d7d-4fmb9\" (UID: \"448e3488-54f3-43e2-800e-de6b74a9b4f8\") " pod="openshift-ingress/router-default-8584f4d7d-4fmb9"
Sep 16 17:27:18.626473 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:18.626477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/448e3488-54f3-43e2-800e-de6b74a9b4f8-metrics-certs\") pod \"router-default-8584f4d7d-4fmb9\" (UID: \"448e3488-54f3-43e2-800e-de6b74a9b4f8\") " pod="openshift-ingress/router-default-8584f4d7d-4fmb9"
Sep 16 17:27:18.627148 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:18.627123    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/448e3488-54f3-43e2-800e-de6b74a9b4f8-service-ca-bundle\") pod \"router-default-8584f4d7d-4fmb9\" (UID: \"448e3488-54f3-43e2-800e-de6b74a9b4f8\") " pod="openshift-ingress/router-default-8584f4d7d-4fmb9"
Sep 16 17:27:18.628821 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:18.628803    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/448e3488-54f3-43e2-800e-de6b74a9b4f8-metrics-certs\") pod \"router-default-8584f4d7d-4fmb9\" (UID: \"448e3488-54f3-43e2-800e-de6b74a9b4f8\") " pod="openshift-ingress/router-default-8584f4d7d-4fmb9"
Sep 16 17:27:18.707041 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:18.706948    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8584f4d7d-4fmb9"
Sep 16 17:27:18.929446 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:18.929414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f9d6a0b7-d55d-4919-b951-eb073d89c853-registry-tls\") pod \"image-registry-7d66484d65-bqvc5\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") " pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:18.931733 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:18.931715    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f9d6a0b7-d55d-4919-b951-eb073d89c853-registry-tls\") pod \"image-registry-7d66484d65-bqvc5\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") " pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:18.980370 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:18.980300    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8584f4d7d-4fmb9"]
Sep 16 17:27:18.980370 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:18.980349    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:18.982984 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:27:18.982961    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod448e3488_54f3_43e2_800e_de6b74a9b4f8.slice/crio-9f5f1b02297fc6ea6f263551d3bb6999f469202a1921675904f546e137700b60 WatchSource:0}: Error finding container 9f5f1b02297fc6ea6f263551d3bb6999f469202a1921675904f546e137700b60: Status 404 returned error can't find the container with id 9f5f1b02297fc6ea6f263551d3bb6999f469202a1921675904f546e137700b60
Sep 16 17:27:19.022856 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:19.022827    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:19.260357 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:19.260289    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d66484d65-bqvc5"]
Sep 16 17:27:19.263181 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:27:19.263148    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d6a0b7_d55d_4919_b951_eb073d89c853.slice/crio-e02c6ddf0e1d38888d0f9eb55986f1777ac125f5b62f13f26053aa6a38bc438a WatchSource:0}: Error finding container e02c6ddf0e1d38888d0f9eb55986f1777ac125f5b62f13f26053aa6a38bc438a: Status 404 returned error can't find the container with id e02c6ddf0e1d38888d0f9eb55986f1777ac125f5b62f13f26053aa6a38bc438a
Sep 16 17:27:19.635330 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:19.635293    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:19.735512 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:19.735457    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8584f4d7d-4fmb9" event={"ID":"448e3488-54f3-43e2-800e-de6b74a9b4f8","Type":"ContainerStarted","Data":"e92c716b648248f94959b86f42f7c8900b3e07702f1ab91ac548cd94fc5c4ab4"}
Sep 16 17:27:19.735689 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:19.735528    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8584f4d7d-4fmb9" event={"ID":"448e3488-54f3-43e2-800e-de6b74a9b4f8","Type":"ContainerStarted","Data":"9f5f1b02297fc6ea6f263551d3bb6999f469202a1921675904f546e137700b60"}
Sep 16 17:27:19.736922 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:19.736897    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d66484d65-bqvc5" event={"ID":"f9d6a0b7-d55d-4919-b951-eb073d89c853","Type":"ContainerStarted","Data":"962dfd22cb93c785359251f933914c8dcbff8236635653496d92f5c38e2cc368"}
Sep 16 17:27:19.737033 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:19.736935    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d66484d65-bqvc5" event={"ID":"f9d6a0b7-d55d-4919-b951-eb073d89c853","Type":"ContainerStarted","Data":"e02c6ddf0e1d38888d0f9eb55986f1777ac125f5b62f13f26053aa6a38bc438a"}
Sep 16 17:27:19.737091 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:19.737056    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:19.758382 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:19.758340    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-8584f4d7d-4fmb9" podStartSLOduration=17.75832676 podStartE2EDuration="17.75832676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:19.757343076 +0000 UTC m=+161.120021863" watchObservedRunningTime="2026-09-16 17:27:19.75832676 +0000 UTC m=+161.121005546"
Sep 16 17:27:19.782579 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:19.782542    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7d66484d65-bqvc5" podStartSLOduration=16.782529117 podStartE2EDuration="16.782529117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:19.781607661 +0000 UTC m=+161.144286447" watchObservedRunningTime="2026-09-16 17:27:19.782529117 +0000 UTC m=+161.145207902"
Sep 16 17:27:19.820134 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:19.820089    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:20.000215 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:20.000130    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:20.220584 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:20.220551    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:20.434358 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:20.434320    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:20.653691 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:20.653655    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:20.707661 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:20.707590    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-8584f4d7d-4fmb9"
Sep 16 17:27:20.710228 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:20.710205    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-8584f4d7d-4fmb9"
Sep 16 17:27:20.741648 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:20.741615    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-8584f4d7d-4fmb9"
Sep 16 17:27:20.742764 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:20.742742    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-8584f4d7d-4fmb9"
Sep 16 17:27:21.427314 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:21.427278    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:21.623244 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:21.623211    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:22.141060 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:22.141026    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:22.246767 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:22.246736    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:22.402274 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:22.402192    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:22.456720 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:22.456684    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34b1ddc8-ea48-4109-852d-24db0edad32c-metrics-tls\") pod \"dns-default-dbvzj\" (UID: \"34b1ddc8-ea48-4109-852d-24db0edad32c\") " pod="openshift-dns/dns-default-dbvzj"
Sep 16 17:27:22.458941 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:22.458918    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34b1ddc8-ea48-4109-852d-24db0edad32c-metrics-tls\") pod \"dns-default-dbvzj\" (UID: \"34b1ddc8-ea48-4109-852d-24db0edad32c\") " pod="openshift-dns/dns-default-dbvzj"
Sep 16 17:27:22.532052 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:22.532023    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qld9l\""
Sep 16 17:27:22.539518 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:22.539492    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dbvzj"
Sep 16 17:27:22.547482 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:22.547449    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:22.772001 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:22.771892    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:22.822677 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:22.822643    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dbvzj"]
Sep 16 17:27:22.825832 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:27:22.825807    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b1ddc8_ea48_4109_852d_24db0edad32c.slice/crio-f5c102c5251c159528cccec119f102a51fedaae024edc2f0f572f7e59fd20ba7 WatchSource:0}: Error finding container f5c102c5251c159528cccec119f102a51fedaae024edc2f0f572f7e59fd20ba7: Status 404 returned error can't find the container with id f5c102c5251c159528cccec119f102a51fedaae024edc2f0f572f7e59fd20ba7
Sep 16 17:27:23.262900 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:23.262843    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad27164e-ba4c-429e-a321-67e56945b867-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-49fvb\" (UID: \"ad27164e-ba4c-429e-a321-67e56945b867\") " pod="openshift-insights/insights-runtime-extractor-49fvb"
Sep 16 17:27:23.263328 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:23.262916    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad27164e-ba4c-429e-a321-67e56945b867-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-49fvb\" (UID: \"ad27164e-ba4c-429e-a321-67e56945b867\") " pod="openshift-insights/insights-runtime-extractor-49fvb"
Sep 16 17:27:23.263488 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:23.263464    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad27164e-ba4c-429e-a321-67e56945b867-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-49fvb\" (UID: \"ad27164e-ba4c-429e-a321-67e56945b867\") " pod="openshift-insights/insights-runtime-extractor-49fvb"
Sep 16 17:27:23.265592 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:23.265568    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad27164e-ba4c-429e-a321-67e56945b867-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-49fvb\" (UID: \"ad27164e-ba4c-429e-a321-67e56945b867\") " pod="openshift-insights/insights-runtime-extractor-49fvb"
Sep 16 17:27:23.338217 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:23.338177    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-49fvb"
Sep 16 17:27:23.615996 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:23.615557    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-49fvb"]
Sep 16 17:27:23.618658 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:27:23.618630    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad27164e_ba4c_429e_a321_67e56945b867.slice/crio-58947558a3c0740ba9dd5f0496f823ebebcad994ccdeb1824e2b3498a4238543 WatchSource:0}: Error finding container 58947558a3c0740ba9dd5f0496f823ebebcad994ccdeb1824e2b3498a4238543: Status 404 returned error can't find the container with id 58947558a3c0740ba9dd5f0496f823ebebcad994ccdeb1824e2b3498a4238543
Sep 16 17:27:23.753032 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:23.752997    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-49fvb" event={"ID":"ad27164e-ba4c-429e-a321-67e56945b867","Type":"ContainerStarted","Data":"f42606936557585ed0209f318b98bc886462e2dc1f01e2a8af21d786f000b2b7"}
Sep 16 17:27:23.753185 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:23.753055    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-49fvb" event={"ID":"ad27164e-ba4c-429e-a321-67e56945b867","Type":"ContainerStarted","Data":"58947558a3c0740ba9dd5f0496f823ebebcad994ccdeb1824e2b3498a4238543"}
Sep 16 17:27:23.754068 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:23.754041    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dbvzj" event={"ID":"34b1ddc8-ea48-4109-852d-24db0edad32c","Type":"ContainerStarted","Data":"f5c102c5251c159528cccec119f102a51fedaae024edc2f0f572f7e59fd20ba7"}
Sep 16 17:27:24.759167 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:24.759071    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dbvzj" event={"ID":"34b1ddc8-ea48-4109-852d-24db0edad32c","Type":"ContainerStarted","Data":"2466b6c9496aa73ab4c062eef89241ae6789af4370d2050598717eb44a238cfb"}
Sep 16 17:27:24.759167 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:24.759134    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dbvzj" event={"ID":"34b1ddc8-ea48-4109-852d-24db0edad32c","Type":"ContainerStarted","Data":"bae101298f33acfcf6738c1d7d815148e200117e4cd5b4d9ab1ad06146efca2a"}
Sep 16 17:27:24.759167 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:24.759168    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dbvzj"
Sep 16 17:27:24.760898 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:24.760856    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-49fvb" event={"ID":"ad27164e-ba4c-429e-a321-67e56945b867","Type":"ContainerStarted","Data":"e41c2fb52f642f9b564c56a7e9e6e05ea418805851dc10bda303d8aabe9574dc"}
Sep 16 17:27:24.781038 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:24.780990    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dbvzj" podStartSLOduration=129.361056984 podStartE2EDuration="2m10.780975074s" totalImagesPullingTime="1.41991809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:25:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:24.780790666 +0000 UTC m=+166.143469451" watchObservedRunningTime="2026-09-16 17:27:24.780975074 +0000 UTC m=+166.143653851"
Sep 16 17:27:26.770112 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:26.770075    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-49fvb" event={"ID":"ad27164e-ba4c-429e-a321-67e56945b867","Type":"ContainerStarted","Data":"ee61f3a8681639c396af85289eb2431772169577654292ebe185f84a7535412e"}
Sep 16 17:27:27.604434 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:27.604384    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:27.747495 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:27.747461    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:27.955502 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:27.955406    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:28.243466 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:28.243384    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:28.397609 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:28.397574    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:28.525166 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:28.525086    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:28.697017 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:28.696987    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:29.218704 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:29.218663    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:29.341913 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:29.341858    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:29.443737 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:29.443702    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:30.461969 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:30.461936    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:31.078226 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:31.078194    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:27:31.861323 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:31.861279    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-49fvb" podStartSLOduration=22.787560155 podStartE2EDuration="24.861262052s" totalImagesPullingTime="2.073701897s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:27:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:26.801182433 +0000 UTC m=+168.163861220" watchObservedRunningTime="2026-09-16 17:27:31.861262052 +0000 UTC m=+173.223940837"
Sep 16 17:27:31.862524 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:31.862500    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7d66484d65-bqvc5"]
Sep 16 17:27:31.866974 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:31.866951    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-z2f4v"]
Sep 16 17:27:31.869807 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:31.869792    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-z2f4v"
Sep 16 17:27:31.893763 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:31.893742    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 17:27:31.894050 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:31.894035    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-dkhvk\""
Sep 16 17:27:31.900121 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:31.900105    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 17:27:31.907675 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:31.907655    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-z2f4v"]
Sep 16 17:27:31.927530 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:31.927506    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rj2b\" (UniqueName: \"kubernetes.io/projected/bbb73d75-b302-4ef0-b3ef-01a02b6807ba-kube-api-access-9rj2b\") pod \"downloads-76f99584f4-z2f4v\" (UID: \"bbb73d75-b302-4ef0-b3ef-01a02b6807ba\") " pod="openshift-console/downloads-76f99584f4-z2f4v"
Sep 16 17:27:32.006982 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.006951    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-75c5879656-qjbs4"]
Sep 16 17:27:32.009897 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.009881    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75c5879656-qjbs4"
Sep 16 17:27:32.028662 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.028639    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02f4fa27-fb17-4f20-9500-f2d24f08a254-registry-tls\") pod \"image-registry-75c5879656-qjbs4\" (UID: \"02f4fa27-fb17-4f20-9500-f2d24f08a254\") " pod="openshift-image-registry/image-registry-75c5879656-qjbs4"
Sep 16 17:27:32.028752 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.028668    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/02f4fa27-fb17-4f20-9500-f2d24f08a254-registry-certificates\") pod \"image-registry-75c5879656-qjbs4\" (UID: \"02f4fa27-fb17-4f20-9500-f2d24f08a254\") " pod="openshift-image-registry/image-registry-75c5879656-qjbs4"
Sep 16 17:27:32.028752 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.028691    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/02f4fa27-fb17-4f20-9500-f2d24f08a254-trusted-ca\") pod \"image-registry-75c5879656-qjbs4\" (UID: \"02f4fa27-fb17-4f20-9500-f2d24f08a254\") " pod="openshift-image-registry/image-registry-75c5879656-qjbs4"
Sep 16 17:27:32.028830 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.028744    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/02f4fa27-fb17-4f20-9500-f2d24f08a254-image-registry-private-configuration\") pod \"image-registry-75c5879656-qjbs4\" (UID: \"02f4fa27-fb17-4f20-9500-f2d24f08a254\") " pod="openshift-image-registry/image-registry-75c5879656-qjbs4"
Sep 16 17:27:32.028830 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.028779    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/02f4fa27-fb17-4f20-9500-f2d24f08a254-installation-pull-secrets\") pod \"image-registry-75c5879656-qjbs4\" (UID: \"02f4fa27-fb17-4f20-9500-f2d24f08a254\") " pod="openshift-image-registry/image-registry-75c5879656-qjbs4"
Sep 16 17:27:32.028830 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.028806    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02f4fa27-fb17-4f20-9500-f2d24f08a254-bound-sa-token\") pod \"image-registry-75c5879656-qjbs4\" (UID: \"02f4fa27-fb17-4f20-9500-f2d24f08a254\") " pod="openshift-image-registry/image-registry-75c5879656-qjbs4"
Sep 16 17:27:32.028933 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.028845    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pqgl\" (UniqueName: \"kubernetes.io/projected/02f4fa27-fb17-4f20-9500-f2d24f08a254-kube-api-access-2pqgl\") pod \"image-registry-75c5879656-qjbs4\" (UID: \"02f4fa27-fb17-4f20-9500-f2d24f08a254\") " pod="openshift-image-registry/image-registry-75c5879656-qjbs4"
Sep 16 17:27:32.028933 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.028914    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rj2b\" (UniqueName: \"kubernetes.io/projected/bbb73d75-b302-4ef0-b3ef-01a02b6807ba-kube-api-access-9rj2b\") pod \"downloads-76f99584f4-z2f4v\" (UID: \"bbb73d75-b302-4ef0-b3ef-01a02b6807ba\") " pod="openshift-console/downloads-76f99584f4-z2f4v"
Sep 16 17:27:32.028995 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.028943    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/02f4fa27-fb17-4f20-9500-f2d24f08a254-ca-trust-extracted\") pod \"image-registry-75c5879656-qjbs4\" (UID: \"02f4fa27-fb17-4f20-9500-f2d24f08a254\") " pod="openshift-image-registry/image-registry-75c5879656-qjbs4"
Sep 16 17:27:32.035781 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.035759    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75c5879656-qjbs4"]
Sep 16 17:27:32.043497 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.043470    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rj2b\" (UniqueName: \"kubernetes.io/projected/bbb73d75-b302-4ef0-b3ef-01a02b6807ba-kube-api-access-9rj2b\") pod \"downloads-76f99584f4-z2f4v\" (UID: \"bbb73d75-b302-4ef0-b3ef-01a02b6807ba\") " pod="openshift-console/downloads-76f99584f4-z2f4v"
Sep 16 17:27:32.108947 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.108882    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:32.129843 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.129817    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/02f4fa27-fb17-4f20-9500-f2d24f08a254-ca-trust-extracted\") pod \"image-registry-75c5879656-qjbs4\" (UID: \"02f4fa27-fb17-4f20-9500-f2d24f08a254\") " pod="openshift-image-registry/image-registry-75c5879656-qjbs4"
Sep 16 17:27:32.129960 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.129855    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02f4fa27-fb17-4f20-9500-f2d24f08a254-registry-tls\") pod \"image-registry-75c5879656-qjbs4\" (UID: \"02f4fa27-fb17-4f20-9500-f2d24f08a254\") " pod="openshift-image-registry/image-registry-75c5879656-qjbs4"
Sep 16 17:27:32.129960 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.129897    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/02f4fa27-fb17-4f20-9500-f2d24f08a254-registry-certificates\") pod \"image-registry-75c5879656-qjbs4\" (UID: \"02f4fa27-fb17-4f20-9500-f2d24f08a254\") " pod="openshift-image-registry/image-registry-75c5879656-qjbs4"
Sep 16 17:27:32.129960 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.129926    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/02f4fa27-fb17-4f20-9500-f2d24f08a254-trusted-ca\") pod \"image-registry-75c5879656-qjbs4\" (UID: \"02f4fa27-fb17-4f20-9500-f2d24f08a254\") " pod="openshift-image-registry/image-registry-75c5879656-qjbs4"
Sep 16 17:27:32.130117 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.129962    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/02f4fa27-fb17-4f20-9500-f2d24f08a254-image-registry-private-configuration\") pod \"image-registry-75c5879656-qjbs4\" (UID: \"02f4fa27-fb17-4f20-9500-f2d24f08a254\") " pod="openshift-image-registry/image-registry-75c5879656-qjbs4"
Sep 16 17:27:32.130117 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.130030    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/02f4fa27-fb17-4f20-9500-f2d24f08a254-installation-pull-secrets\") pod \"image-registry-75c5879656-qjbs4\" (UID: \"02f4fa27-fb17-4f20-9500-f2d24f08a254\") " pod="openshift-image-registry/image-registry-75c5879656-qjbs4"
Sep 16 17:27:32.130117 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.130081    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02f4fa27-fb17-4f20-9500-f2d24f08a254-bound-sa-token\") pod \"image-registry-75c5879656-qjbs4\" (UID: \"02f4fa27-fb17-4f20-9500-f2d24f08a254\") " pod="openshift-image-registry/image-registry-75c5879656-qjbs4"
Sep 16 17:27:32.130259 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.130151    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pqgl\" (UniqueName: \"kubernetes.io/projected/02f4fa27-fb17-4f20-9500-f2d24f08a254-kube-api-access-2pqgl\") pod \"image-registry-75c5879656-qjbs4\" (UID: \"02f4fa27-fb17-4f20-9500-f2d24f08a254\") " pod="openshift-image-registry/image-registry-75c5879656-qjbs4"
Sep 16 17:27:32.130259 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.130202    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/02f4fa27-fb17-4f20-9500-f2d24f08a254-ca-trust-extracted\") pod \"image-registry-75c5879656-qjbs4\" (UID: \"02f4fa27-fb17-4f20-9500-f2d24f08a254\") " pod="openshift-image-registry/image-registry-75c5879656-qjbs4"
Sep 16 17:27:32.130934 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.130906    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/02f4fa27-fb17-4f20-9500-f2d24f08a254-registry-certificates\") pod \"image-registry-75c5879656-qjbs4\" (UID: \"02f4fa27-fb17-4f20-9500-f2d24f08a254\") " pod="openshift-image-registry/image-registry-75c5879656-qjbs4"
Sep 16 17:27:32.131129 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.131113    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/02f4fa27-fb17-4f20-9500-f2d24f08a254-trusted-ca\") pod \"image-registry-75c5879656-qjbs4\" (UID: \"02f4fa27-fb17-4f20-9500-f2d24f08a254\") " pod="openshift-image-registry/image-registry-75c5879656-qjbs4"
Sep 16 17:27:32.132411 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.132384    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02f4fa27-fb17-4f20-9500-f2d24f08a254-registry-tls\") pod \"image-registry-75c5879656-qjbs4\" (UID: \"02f4fa27-fb17-4f20-9500-f2d24f08a254\") " pod="openshift-image-registry/image-registry-75c5879656-qjbs4"
Sep 16 17:27:32.132518 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.132495    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/02f4fa27-fb17-4f20-9500-f2d24f08a254-installation-pull-secrets\") pod \"image-registry-75c5879656-qjbs4\" (UID: \"02f4fa27-fb17-4f20-9500-f2d24f08a254\") " pod="openshift-image-registry/image-registry-75c5879656-qjbs4"
Sep 16 17:27:32.132564 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.132541    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/02f4fa27-fb17-4f20-9500-f2d24f08a254-image-registry-private-configuration\") pod \"image-registry-75c5879656-qjbs4\" (UID: \"02f4fa27-fb17-4f20-9500-f2d24f08a254\") " pod="openshift-image-registry/image-registry-75c5879656-qjbs4"
Sep 16 17:27:32.142829 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.142797    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02f4fa27-fb17-4f20-9500-f2d24f08a254-bound-sa-token\") pod \"image-registry-75c5879656-qjbs4\" (UID: \"02f4fa27-fb17-4f20-9500-f2d24f08a254\") " pod="openshift-image-registry/image-registry-75c5879656-qjbs4"
Sep 16 17:27:32.142911 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.142896    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pqgl\" (UniqueName: \"kubernetes.io/projected/02f4fa27-fb17-4f20-9500-f2d24f08a254-kube-api-access-2pqgl\") pod \"image-registry-75c5879656-qjbs4\" (UID: \"02f4fa27-fb17-4f20-9500-f2d24f08a254\") " pod="openshift-image-registry/image-registry-75c5879656-qjbs4"
Sep 16 17:27:32.177880 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.177838    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-z2f4v"
Sep 16 17:27:32.222273 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.221897    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:32.317904 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.317850    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75c5879656-qjbs4"
Sep 16 17:27:32.458740 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.458710    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-z2f4v"]
Sep 16 17:27:32.460909 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:27:32.460882    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbb73d75_b302_4ef0_b3ef_01a02b6807ba.slice/crio-84d21a49d4f51ad38bad36a5f59574c0c434b7fa78af8af285a31b164119ef29 WatchSource:0}: Error finding container 84d21a49d4f51ad38bad36a5f59574c0c434b7fa78af8af285a31b164119ef29: Status 404 returned error can't find the container with id 84d21a49d4f51ad38bad36a5f59574c0c434b7fa78af8af285a31b164119ef29
Sep 16 17:27:32.593977 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.593941    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75c5879656-qjbs4"]
Sep 16 17:27:32.596552 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:27:32.596523    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f4fa27_fb17_4f20_9500_f2d24f08a254.slice/crio-c3350cb902ad3aa0064ffa7e7c5b95ee7656e6f8a52d18f1a465db7005ce8298 WatchSource:0}: Error finding container c3350cb902ad3aa0064ffa7e7c5b95ee7656e6f8a52d18f1a465db7005ce8298: Status 404 returned error can't find the container with id c3350cb902ad3aa0064ffa7e7c5b95ee7656e6f8a52d18f1a465db7005ce8298
Sep 16 17:27:32.792701 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.792662    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75c5879656-qjbs4" event={"ID":"02f4fa27-fb17-4f20-9500-f2d24f08a254","Type":"ContainerStarted","Data":"b9f5812ae670c69a38be8c6a4f6494c6d0d552e4a4efec4bca7981b771571190"}
Sep 16 17:27:32.792912 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.792715    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75c5879656-qjbs4" event={"ID":"02f4fa27-fb17-4f20-9500-f2d24f08a254","Type":"ContainerStarted","Data":"c3350cb902ad3aa0064ffa7e7c5b95ee7656e6f8a52d18f1a465db7005ce8298"}
Sep 16 17:27:32.792912 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.792764    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-75c5879656-qjbs4"
Sep 16 17:27:32.793958 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.793933    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-z2f4v" event={"ID":"bbb73d75-b302-4ef0-b3ef-01a02b6807ba","Type":"ContainerStarted","Data":"84d21a49d4f51ad38bad36a5f59574c0c434b7fa78af8af285a31b164119ef29"}
Sep 16 17:27:32.834119 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:32.834054    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-75c5879656-qjbs4" podStartSLOduration=1.8340302990000001 podStartE2EDuration="1.834030299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:32.832682017 +0000 UTC m=+174.195360804" watchObservedRunningTime="2026-09-16 17:27:32.834030299 +0000 UTC m=+174.196709089"
Sep 16 17:27:34.533136 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:34.533091    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:34.683853 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:34.683805    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:34.768161 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:34.768131    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dbvzj"
Sep 16 17:27:35.264320 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:35.264279    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:35.364472 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:35.364440    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:36.901009 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:36.900941    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:38.315728 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:38.315688    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:38.543366 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:38.543331    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:38.794727 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:38.794641    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:38.986516 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:38.986468    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:39.426086 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.426044    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:39.547536 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.547504    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7649p"]
Sep 16 17:27:39.550938 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.550914    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7649p"
Sep 16 17:27:39.558667 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.558603    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 17:27:39.559828 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.559670    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 17:27:39.559828 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.559757    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 17:27:39.560019 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.559885    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 17:27:39.560081 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.560018    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gvkfk\""
Sep 16 17:27:39.560815 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.560792    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 17:27:39.560940 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.560838    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 17:27:39.593518 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.593489    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dztcc\" (UniqueName: \"kubernetes.io/projected/d617a6d5-32c8-4728-b616-b7872e50019e-kube-api-access-dztcc\") pod \"node-exporter-7649p\" (UID: \"d617a6d5-32c8-4728-b616-b7872e50019e\") " pod="openshift-monitoring/node-exporter-7649p"
Sep 16 17:27:39.593647 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.593527    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d617a6d5-32c8-4728-b616-b7872e50019e-node-exporter-textfile\") pod \"node-exporter-7649p\" (UID: \"d617a6d5-32c8-4728-b616-b7872e50019e\") " pod="openshift-monitoring/node-exporter-7649p"
Sep 16 17:27:39.593647 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.593552    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d617a6d5-32c8-4728-b616-b7872e50019e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7649p\" (UID: \"d617a6d5-32c8-4728-b616-b7872e50019e\") " pod="openshift-monitoring/node-exporter-7649p"
Sep 16 17:27:39.593760 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.593638    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d617a6d5-32c8-4728-b616-b7872e50019e-sys\") pod \"node-exporter-7649p\" (UID: \"d617a6d5-32c8-4728-b616-b7872e50019e\") " pod="openshift-monitoring/node-exporter-7649p"
Sep 16 17:27:39.593760 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.593695    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d617a6d5-32c8-4728-b616-b7872e50019e-root\") pod \"node-exporter-7649p\" (UID: \"d617a6d5-32c8-4728-b616-b7872e50019e\") " pod="openshift-monitoring/node-exporter-7649p"
Sep 16 17:27:39.593760 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.593720    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d617a6d5-32c8-4728-b616-b7872e50019e-node-exporter-accelerators-collector-config\") pod \"node-exporter-7649p\" (UID: \"d617a6d5-32c8-4728-b616-b7872e50019e\") " pod="openshift-monitoring/node-exporter-7649p"
Sep 16 17:27:39.593937 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.593783    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d617a6d5-32c8-4728-b616-b7872e50019e-node-exporter-wtmp\") pod \"node-exporter-7649p\" (UID: \"d617a6d5-32c8-4728-b616-b7872e50019e\") " pod="openshift-monitoring/node-exporter-7649p"
Sep 16 17:27:39.593937 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.593808    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d617a6d5-32c8-4728-b616-b7872e50019e-metrics-client-ca\") pod \"node-exporter-7649p\" (UID: \"d617a6d5-32c8-4728-b616-b7872e50019e\") " pod="openshift-monitoring/node-exporter-7649p"
Sep 16 17:27:39.593937 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.593839    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d617a6d5-32c8-4728-b616-b7872e50019e-node-exporter-tls\") pod \"node-exporter-7649p\" (UID: \"d617a6d5-32c8-4728-b616-b7872e50019e\") " pod="openshift-monitoring/node-exporter-7649p"
Sep 16 17:27:39.694721 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.694642    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d617a6d5-32c8-4728-b616-b7872e50019e-node-exporter-textfile\") pod \"node-exporter-7649p\" (UID: \"d617a6d5-32c8-4728-b616-b7872e50019e\") " pod="openshift-monitoring/node-exporter-7649p"
Sep 16 17:27:39.694721 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.694676    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d617a6d5-32c8-4728-b616-b7872e50019e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7649p\" (UID: \"d617a6d5-32c8-4728-b616-b7872e50019e\") " pod="openshift-monitoring/node-exporter-7649p"
Sep 16 17:27:39.694973 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.694728    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d617a6d5-32c8-4728-b616-b7872e50019e-sys\") pod \"node-exporter-7649p\" (UID: \"d617a6d5-32c8-4728-b616-b7872e50019e\") " pod="openshift-monitoring/node-exporter-7649p"
Sep 16 17:27:39.694973 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.694772    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d617a6d5-32c8-4728-b616-b7872e50019e-root\") pod \"node-exporter-7649p\" (UID: \"d617a6d5-32c8-4728-b616-b7872e50019e\") " pod="openshift-monitoring/node-exporter-7649p"
Sep 16 17:27:39.694973 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.694796    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d617a6d5-32c8-4728-b616-b7872e50019e-node-exporter-accelerators-collector-config\") pod \"node-exporter-7649p\" (UID: \"d617a6d5-32c8-4728-b616-b7872e50019e\") " pod="openshift-monitoring/node-exporter-7649p"
Sep 16 17:27:39.694973 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.694844    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d617a6d5-32c8-4728-b616-b7872e50019e-node-exporter-wtmp\") pod \"node-exporter-7649p\" (UID: \"d617a6d5-32c8-4728-b616-b7872e50019e\") " pod="openshift-monitoring/node-exporter-7649p"
Sep 16 17:27:39.694973 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.694892    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d617a6d5-32c8-4728-b616-b7872e50019e-metrics-client-ca\") pod \"node-exporter-7649p\" (UID: \"d617a6d5-32c8-4728-b616-b7872e50019e\") " pod="openshift-monitoring/node-exporter-7649p"
Sep 16 17:27:39.694973 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.694927    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d617a6d5-32c8-4728-b616-b7872e50019e-node-exporter-tls\") pod \"node-exporter-7649p\" (UID: \"d617a6d5-32c8-4728-b616-b7872e50019e\") " pod="openshift-monitoring/node-exporter-7649p"
Sep 16 17:27:39.694973 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.694959    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dztcc\" (UniqueName: \"kubernetes.io/projected/d617a6d5-32c8-4728-b616-b7872e50019e-kube-api-access-dztcc\") pod \"node-exporter-7649p\" (UID: \"d617a6d5-32c8-4728-b616-b7872e50019e\") " pod="openshift-monitoring/node-exporter-7649p"
Sep 16 17:27:39.695315 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.694887    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d617a6d5-32c8-4728-b616-b7872e50019e-sys\") pod \"node-exporter-7649p\" (UID: \"d617a6d5-32c8-4728-b616-b7872e50019e\") " pod="openshift-monitoring/node-exporter-7649p"
Sep 16 17:27:39.695315 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.695042    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d617a6d5-32c8-4728-b616-b7872e50019e-node-exporter-textfile\") pod \"node-exporter-7649p\" (UID: \"d617a6d5-32c8-4728-b616-b7872e50019e\") " pod="openshift-monitoring/node-exporter-7649p"
Sep 16 17:27:39.695315 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.695050    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d617a6d5-32c8-4728-b616-b7872e50019e-root\") pod \"node-exporter-7649p\" (UID: \"d617a6d5-32c8-4728-b616-b7872e50019e\") " pod="openshift-monitoring/node-exporter-7649p"
Sep 16 17:27:39.695315 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.695178    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d617a6d5-32c8-4728-b616-b7872e50019e-node-exporter-wtmp\") pod \"node-exporter-7649p\" (UID: \"d617a6d5-32c8-4728-b616-b7872e50019e\") " pod="openshift-monitoring/node-exporter-7649p"
Sep 16 17:27:39.695582 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.695561    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d617a6d5-32c8-4728-b616-b7872e50019e-metrics-client-ca\") pod \"node-exporter-7649p\" (UID: \"d617a6d5-32c8-4728-b616-b7872e50019e\") " pod="openshift-monitoring/node-exporter-7649p"
Sep 16 17:27:39.695691 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.695658    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d617a6d5-32c8-4728-b616-b7872e50019e-node-exporter-accelerators-collector-config\") pod \"node-exporter-7649p\" (UID: \"d617a6d5-32c8-4728-b616-b7872e50019e\") " pod="openshift-monitoring/node-exporter-7649p"
Sep 16 17:27:39.697367 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.697344    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d617a6d5-32c8-4728-b616-b7872e50019e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7649p\" (UID: \"d617a6d5-32c8-4728-b616-b7872e50019e\") " pod="openshift-monitoring/node-exporter-7649p"
Sep 16 17:27:39.697471 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.697408    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d617a6d5-32c8-4728-b616-b7872e50019e-node-exporter-tls\") pod \"node-exporter-7649p\" (UID: \"d617a6d5-32c8-4728-b616-b7872e50019e\") " pod="openshift-monitoring/node-exporter-7649p"
Sep 16 17:27:39.726216 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.726191    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dztcc\" (UniqueName: \"kubernetes.io/projected/d617a6d5-32c8-4728-b616-b7872e50019e-kube-api-access-dztcc\") pod \"node-exporter-7649p\" (UID: \"d617a6d5-32c8-4728-b616-b7872e50019e\") " pod="openshift-monitoring/node-exporter-7649p"
Sep 16 17:27:39.782495 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.782452    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:39.862544 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:39.862510    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7649p"
Sep 16 17:27:39.874256 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:27:39.874231    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd617a6d5_32c8_4728_b616_b7872e50019e.slice/crio-715ab315cbf1ec9baef36c04a49cfc327bab3e0c494d5ad2bf569b0db3815f3f WatchSource:0}: Error finding container 715ab315cbf1ec9baef36c04a49cfc327bab3e0c494d5ad2bf569b0db3815f3f: Status 404 returned error can't find the container with id 715ab315cbf1ec9baef36c04a49cfc327bab3e0c494d5ad2bf569b0db3815f3f
Sep 16 17:27:40.121235 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:40.121201    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:40.233077 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:40.233038    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:40.354812 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:40.354758    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:40.824986 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:40.824947    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7649p" event={"ID":"d617a6d5-32c8-4728-b616-b7872e50019e","Type":"ContainerStarted","Data":"715ab315cbf1ec9baef36c04a49cfc327bab3e0c494d5ad2bf569b0db3815f3f"}
Sep 16 17:27:40.968118 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:40.967998    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:41.090335 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:41.090235    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:41.830180 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:41.830144    2577 generic.go:356] "Generic (PLEG): container finished" podID="d617a6d5-32c8-4728-b616-b7872e50019e" containerID="022a5d05f14d09bd75281e3ae7aadf9b63dffa707e1929976b1992e47b062aaa" exitCode=0
Sep 16 17:27:41.830595 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:41.830186    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7649p" event={"ID":"d617a6d5-32c8-4728-b616-b7872e50019e","Type":"ContainerDied","Data":"022a5d05f14d09bd75281e3ae7aadf9b63dffa707e1929976b1992e47b062aaa"}
Sep 16 17:27:41.868358 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:41.868326    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:42.068584 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:42.068543    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:42.161802 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:42.161653    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:42.437021 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:42.436916    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:42.615590 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:42.615549    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:43.731129 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:43.731087    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:43.761398 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:43.761368    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-sx5wx"]
Sep 16 17:27:43.764693 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:43.764669    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-sx5wx"
Sep 16 17:27:43.767549 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:43.767498    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 16 17:27:43.767549 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:43.767531    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-tncs4\""
Sep 16 17:27:43.775047 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:43.775023    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-sx5wx"]
Sep 16 17:27:43.824183 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:43.824143    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:43.831054 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:43.831026    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8564b3ae-746c-4d4c-bade-c8b85d78078b-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-sx5wx\" (UID: \"8564b3ae-746c-4d4c-bade-c8b85d78078b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-sx5wx"
Sep 16 17:27:43.831210 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:43.831126    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q27m5\" (UniqueName: \"kubernetes.io/projected/8564b3ae-746c-4d4c-bade-c8b85d78078b-kube-api-access-q27m5\") pod \"monitoring-plugin-5d85778b9d-sx5wx\" (UID: \"8564b3ae-746c-4d4c-bade-c8b85d78078b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-sx5wx"
Sep 16 17:27:43.931897 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:43.931842    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8564b3ae-746c-4d4c-bade-c8b85d78078b-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-sx5wx\" (UID: \"8564b3ae-746c-4d4c-bade-c8b85d78078b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-sx5wx"
Sep 16 17:27:43.932081 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:43.931939    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q27m5\" (UniqueName: \"kubernetes.io/projected/8564b3ae-746c-4d4c-bade-c8b85d78078b-kube-api-access-q27m5\") pod \"monitoring-plugin-5d85778b9d-sx5wx\" (UID: \"8564b3ae-746c-4d4c-bade-c8b85d78078b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-sx5wx"
Sep 16 17:27:43.932081 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:43.932011    2577 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 16 17:27:43.932179 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:43.932083    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8564b3ae-746c-4d4c-bade-c8b85d78078b-monitoring-plugin-cert podName:8564b3ae-746c-4d4c-bade-c8b85d78078b nodeName:}" failed. No retries permitted until 2026-09-16 17:27:44.432067407 +0000 UTC m=+185.794746176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/8564b3ae-746c-4d4c-bade-c8b85d78078b-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-sx5wx" (UID: "8564b3ae-746c-4d4c-bade-c8b85d78078b") : secret "monitoring-plugin-cert" not found
Sep 16 17:27:43.944338 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:43.944304    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q27m5\" (UniqueName: \"kubernetes.io/projected/8564b3ae-746c-4d4c-bade-c8b85d78078b-kube-api-access-q27m5\") pod \"monitoring-plugin-5d85778b9d-sx5wx\" (UID: \"8564b3ae-746c-4d4c-bade-c8b85d78078b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-sx5wx"
Sep 16 17:27:44.050126 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:44.050036    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:44.435754 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:44.435716    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8564b3ae-746c-4d4c-bade-c8b85d78078b-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-sx5wx\" (UID: \"8564b3ae-746c-4d4c-bade-c8b85d78078b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-sx5wx"
Sep 16 17:27:44.438524 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:44.438499    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8564b3ae-746c-4d4c-bade-c8b85d78078b-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-sx5wx\" (UID: \"8564b3ae-746c-4d4c-bade-c8b85d78078b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-sx5wx"
Sep 16 17:27:44.675088 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:44.675045    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-sx5wx"
Sep 16 17:27:45.056991 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:45.056952    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:45.232739 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:45.232702    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:45.526730 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:45.526694    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:45.717095 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:45.717060    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:48.178393 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:48.178310    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-sx5wx"]
Sep 16 17:27:48.181383 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:27:48.181355    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8564b3ae_746c_4d4c_bade_c8b85d78078b.slice/crio-85285e9113d6610ec32c96fb998472d858842df49a8dcd01dd1db406e3f7a3b4 WatchSource:0}: Error finding container 85285e9113d6610ec32c96fb998472d858842df49a8dcd01dd1db406e3f7a3b4: Status 404 returned error can't find the container with id 85285e9113d6610ec32c96fb998472d858842df49a8dcd01dd1db406e3f7a3b4
Sep 16 17:27:48.857085 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:48.857046    2577 generic.go:356] "Generic (PLEG): container finished" podID="3df2e9f8-8583-4672-bde5-56af14edd107" containerID="3b45d0a8ef84d9d97d0504ea19914a41f1c6c1283c99975604ec8f9913b8b91a" exitCode=255
Sep 16 17:27:48.857276 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:48.857122    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fb94778b8-d97jb" event={"ID":"3df2e9f8-8583-4672-bde5-56af14edd107","Type":"ContainerDied","Data":"3b45d0a8ef84d9d97d0504ea19914a41f1c6c1283c99975604ec8f9913b8b91a"}
Sep 16 17:27:48.859150 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:48.859118    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-z2f4v" event={"ID":"bbb73d75-b302-4ef0-b3ef-01a02b6807ba","Type":"ContainerStarted","Data":"aac4161ebc87c83237c279c0b6b42cd44b02022e888a23eb14d0264ff252d2cb"}
Sep 16 17:27:48.859385 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:48.859357    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-z2f4v"
Sep 16 17:27:48.860740 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:48.860691    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-sx5wx" event={"ID":"8564b3ae-746c-4d4c-bade-c8b85d78078b","Type":"ContainerStarted","Data":"85285e9113d6610ec32c96fb998472d858842df49a8dcd01dd1db406e3f7a3b4"}
Sep 16 17:27:48.863416 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:48.863393    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7649p" event={"ID":"d617a6d5-32c8-4728-b616-b7872e50019e","Type":"ContainerStarted","Data":"98bd1bbba4f6eae881b288278df0a06c1311e7866511e7f7f6aed08249477748"}
Sep 16 17:27:48.863504 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:48.863444    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7649p" event={"ID":"d617a6d5-32c8-4728-b616-b7872e50019e","Type":"ContainerStarted","Data":"f3809813193c939debb2ed9c76981159f43a216f102fa9a964509c3e73464e7b"}
Sep 16 17:27:48.865094 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:48.865075    2577 scope.go:117] "RemoveContainer" containerID="3b45d0a8ef84d9d97d0504ea19914a41f1c6c1283c99975604ec8f9913b8b91a"
Sep 16 17:27:48.868569 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:48.868548    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-z2f4v"
Sep 16 17:27:48.969398 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:48.969337    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-z2f4v" podStartSLOduration=2.419905215 podStartE2EDuration="17.969313644s" totalImagesPullingTime="15.549408429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:48.966915541 +0000 UTC m=+190.329594327" watchObservedRunningTime="2026-09-16 17:27:48.969313644 +0000 UTC m=+190.331992431"
Sep 16 17:27:49.009459 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:49.009397    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7649p" podStartSLOduration=8.963436379000001 podStartE2EDuration="10.009375948s" totalImagesPullingTime="974.01342ms" totalInitContainerRuntime="71.926149ms" isStatefulPod=true podCreationTimestamp="2026-09-16 17:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:49.008198696 +0000 UTC m=+190.370877484" watchObservedRunningTime="2026-09-16 17:27:49.009375948 +0000 UTC m=+190.372054734"
Sep 16 17:27:49.046631 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:49.046590    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fb94778b8-d97jb"
Sep 16 17:27:49.869622 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:49.869586    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fb94778b8-d97jb" event={"ID":"3df2e9f8-8583-4672-bde5-56af14edd107","Type":"ContainerStarted","Data":"4554d4cda8ccb50560547ad821270dbd553c7f48e910572db521f3e40f05a708"}
Sep 16 17:27:50.258979 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:50.258938    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:50.401416 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:50.401373    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:50.666304 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:50.666261    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:50.877076 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:50.877039    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-sx5wx" event={"ID":"8564b3ae-746c-4d4c-bade-c8b85d78078b","Type":"ContainerStarted","Data":"ffe3b92b45a40e0e9cc1f962d02d4995366fe626cefa61949e24a29bedb7084f"}
Sep 16 17:27:50.877532 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:50.877486    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-sx5wx"
Sep 16 17:27:50.882626 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:50.882595    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-sx5wx"
Sep 16 17:27:50.900556 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:50.900506    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-sx5wx" podStartSLOduration=6.295067272 podStartE2EDuration="7.900476152s" totalImagesPullingTime="1.60540888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:50.900308696 +0000 UTC m=+192.262987482" watchObservedRunningTime="2026-09-16 17:27:50.900476152 +0000 UTC m=+192.263154937"
Sep 16 17:27:51.079015 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:51.078974    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:51.188902 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:51.188855    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:52.297944 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:52.297890    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:52.658096 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:52.658059    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:53.803491 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:53.803456    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-75c5879656-qjbs4"
Sep 16 17:27:54.029049 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:54.029006    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:54.183271 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:54.183234    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:54.645216 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:54.645180    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:54.886819 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:54.886782    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:27:56.881376 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:56.881338    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7d66484d65-bqvc5" podUID="f9d6a0b7-d55d-4919-b951-eb073d89c853" containerName="registry" containerID="cri-o://962dfd22cb93c785359251f933914c8dcbff8236635653496d92f5c38e2cc368" gracePeriod=30
Sep 16 17:27:57.153119 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.153054    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:57.253084 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.253057    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f9d6a0b7-d55d-4919-b951-eb073d89c853-trusted-ca\") pod \"f9d6a0b7-d55d-4919-b951-eb073d89c853\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") "
Sep 16 17:27:57.253267 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.253093    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpmfs\" (UniqueName: \"kubernetes.io/projected/f9d6a0b7-d55d-4919-b951-eb073d89c853-kube-api-access-fpmfs\") pod \"f9d6a0b7-d55d-4919-b951-eb073d89c853\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") "
Sep 16 17:27:57.253267 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.253119    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f9d6a0b7-d55d-4919-b951-eb073d89c853-registry-certificates\") pod \"f9d6a0b7-d55d-4919-b951-eb073d89c853\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") "
Sep 16 17:27:57.253267 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.253139    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f9d6a0b7-d55d-4919-b951-eb073d89c853-ca-trust-extracted\") pod \"f9d6a0b7-d55d-4919-b951-eb073d89c853\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") "
Sep 16 17:27:57.253267 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.253164    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f9d6a0b7-d55d-4919-b951-eb073d89c853-registry-tls\") pod \"f9d6a0b7-d55d-4919-b951-eb073d89c853\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") "
Sep 16 17:27:57.253267 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.253190    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f9d6a0b7-d55d-4919-b951-eb073d89c853-bound-sa-token\") pod \"f9d6a0b7-d55d-4919-b951-eb073d89c853\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") "
Sep 16 17:27:57.253267 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.253210    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f9d6a0b7-d55d-4919-b951-eb073d89c853-image-registry-private-configuration\") pod \"f9d6a0b7-d55d-4919-b951-eb073d89c853\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") "
Sep 16 17:27:57.253267 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.253251    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f9d6a0b7-d55d-4919-b951-eb073d89c853-installation-pull-secrets\") pod \"f9d6a0b7-d55d-4919-b951-eb073d89c853\" (UID: \"f9d6a0b7-d55d-4919-b951-eb073d89c853\") "
Sep 16 17:27:57.253633 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.253578    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9d6a0b7-d55d-4919-b951-eb073d89c853-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f9d6a0b7-d55d-4919-b951-eb073d89c853" (UID: "f9d6a0b7-d55d-4919-b951-eb073d89c853"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:27:57.253633 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.253596    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9d6a0b7-d55d-4919-b951-eb073d89c853-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f9d6a0b7-d55d-4919-b951-eb073d89c853" (UID: "f9d6a0b7-d55d-4919-b951-eb073d89c853"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:27:57.255766 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.255699    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d6a0b7-d55d-4919-b951-eb073d89c853-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f9d6a0b7-d55d-4919-b951-eb073d89c853" (UID: "f9d6a0b7-d55d-4919-b951-eb073d89c853"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:27:57.255766 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.255702    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9d6a0b7-d55d-4919-b951-eb073d89c853-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f9d6a0b7-d55d-4919-b951-eb073d89c853" (UID: "f9d6a0b7-d55d-4919-b951-eb073d89c853"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:27:57.255969 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.255797    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9d6a0b7-d55d-4919-b951-eb073d89c853-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f9d6a0b7-d55d-4919-b951-eb073d89c853" (UID: "f9d6a0b7-d55d-4919-b951-eb073d89c853"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:27:57.255969 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.255882    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d6a0b7-d55d-4919-b951-eb073d89c853-kube-api-access-fpmfs" (OuterVolumeSpecName: "kube-api-access-fpmfs") pod "f9d6a0b7-d55d-4919-b951-eb073d89c853" (UID: "f9d6a0b7-d55d-4919-b951-eb073d89c853"). InnerVolumeSpecName "kube-api-access-fpmfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:27:57.255969 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.255902    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d6a0b7-d55d-4919-b951-eb073d89c853-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f9d6a0b7-d55d-4919-b951-eb073d89c853" (UID: "f9d6a0b7-d55d-4919-b951-eb073d89c853"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:27:57.261980 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.261956    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d6a0b7-d55d-4919-b951-eb073d89c853-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f9d6a0b7-d55d-4919-b951-eb073d89c853" (UID: "f9d6a0b7-d55d-4919-b951-eb073d89c853"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:27:57.353795 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.353767    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f9d6a0b7-d55d-4919-b951-eb073d89c853-trusted-ca\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:27:57.353795 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.353794    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpmfs\" (UniqueName: \"kubernetes.io/projected/f9d6a0b7-d55d-4919-b951-eb073d89c853-kube-api-access-fpmfs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:27:57.353982 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.353805    2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f9d6a0b7-d55d-4919-b951-eb073d89c853-registry-certificates\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:27:57.353982 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.353814    2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f9d6a0b7-d55d-4919-b951-eb073d89c853-ca-trust-extracted\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:27:57.353982 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.353823    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f9d6a0b7-d55d-4919-b951-eb073d89c853-registry-tls\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:27:57.353982 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.353832    2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f9d6a0b7-d55d-4919-b951-eb073d89c853-bound-sa-token\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:27:57.353982 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.353840    2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f9d6a0b7-d55d-4919-b951-eb073d89c853-image-registry-private-configuration\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:27:57.353982 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.353849    2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f9d6a0b7-d55d-4919-b951-eb073d89c853-installation-pull-secrets\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:27:57.905382 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.905349    2577 generic.go:356] "Generic (PLEG): container finished" podID="f9d6a0b7-d55d-4919-b951-eb073d89c853" containerID="962dfd22cb93c785359251f933914c8dcbff8236635653496d92f5c38e2cc368" exitCode=0
Sep 16 17:27:57.905770 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.905394    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d66484d65-bqvc5"
Sep 16 17:27:57.905770 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.905410    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d66484d65-bqvc5" event={"ID":"f9d6a0b7-d55d-4919-b951-eb073d89c853","Type":"ContainerDied","Data":"962dfd22cb93c785359251f933914c8dcbff8236635653496d92f5c38e2cc368"}
Sep 16 17:27:57.905770 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.905450    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d66484d65-bqvc5" event={"ID":"f9d6a0b7-d55d-4919-b951-eb073d89c853","Type":"ContainerDied","Data":"e02c6ddf0e1d38888d0f9eb55986f1777ac125f5b62f13f26053aa6a38bc438a"}
Sep 16 17:27:57.905770 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.905468    2577 scope.go:117] "RemoveContainer" containerID="962dfd22cb93c785359251f933914c8dcbff8236635653496d92f5c38e2cc368"
Sep 16 17:27:57.914055 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.914037    2577 scope.go:117] "RemoveContainer" containerID="962dfd22cb93c785359251f933914c8dcbff8236635653496d92f5c38e2cc368"
Sep 16 17:27:57.914315 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:27:57.914295    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"962dfd22cb93c785359251f933914c8dcbff8236635653496d92f5c38e2cc368\": container with ID starting with 962dfd22cb93c785359251f933914c8dcbff8236635653496d92f5c38e2cc368 not found: ID does not exist" containerID="962dfd22cb93c785359251f933914c8dcbff8236635653496d92f5c38e2cc368"
Sep 16 17:27:57.914369 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.914325    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"962dfd22cb93c785359251f933914c8dcbff8236635653496d92f5c38e2cc368"} err="failed to get container status \"962dfd22cb93c785359251f933914c8dcbff8236635653496d92f5c38e2cc368\": rpc error: code = NotFound desc = could not find container \"962dfd22cb93c785359251f933914c8dcbff8236635653496d92f5c38e2cc368\": container with ID starting with 962dfd22cb93c785359251f933914c8dcbff8236635653496d92f5c38e2cc368 not found: ID does not exist"
Sep 16 17:27:57.929664 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.929632    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7d66484d65-bqvc5"]
Sep 16 17:27:57.932376 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:57.932352    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7d66484d65-bqvc5"]
Sep 16 17:27:59.079317 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:27:59.079284    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9d6a0b7-d55d-4919-b951-eb073d89c853" path="/var/lib/kubelet/pods/f9d6a0b7-d55d-4919-b951-eb073d89c853/volumes"
Sep 16 17:28:03.909450 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:03.909414    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:04.026759 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:04.026723    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:04.126740 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:04.126706    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:07.049833 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:07.049793    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:07.117829 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:07.117798    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:07.307430 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:07.307349    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:07.435308 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:07.435276    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:08.117324 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:08.117296    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:08.240658 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:08.240618    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:21.981895 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:21.981798    2577 generic.go:356] "Generic (PLEG): container finished" podID="b23b1744-c013-4acf-967c-bfc9972bde31" containerID="b26b68c1a7333ff2fc95a743d3aa0cfa642f15106ea1f65d1b7f2c2ce05c05f7" exitCode=0
Sep 16 17:28:21.981895 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:21.981883    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-6g6v2" event={"ID":"b23b1744-c013-4acf-967c-bfc9972bde31","Type":"ContainerDied","Data":"b26b68c1a7333ff2fc95a743d3aa0cfa642f15106ea1f65d1b7f2c2ce05c05f7"}
Sep 16 17:28:21.982318 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:21.982200    2577 scope.go:117] "RemoveContainer" containerID="b26b68c1a7333ff2fc95a743d3aa0cfa642f15106ea1f65d1b7f2c2ce05c05f7"
Sep 16 17:28:22.290211 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:22.290122    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:22.391133 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:22.391100    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:22.987250 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:22.987216    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-6g6v2" event={"ID":"b23b1744-c013-4acf-967c-bfc9972bde31","Type":"ContainerStarted","Data":"7c3422f2fecff6773aa8245444f169e3126c754b934353df09db627b36c34f98"}
Sep 16 17:28:37.036288 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:37.036255    2577 generic.go:356] "Generic (PLEG): container finished" podID="146f8eb9-491d-4f58-a764-ba70d2197637" containerID="3cd5625bc4788cfd69bbf7c66f183389c78a7bf63d0480aa8b3d44cec5f269e7" exitCode=0
Sep 16 17:28:37.036658 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:37.036303    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jm27w" event={"ID":"146f8eb9-491d-4f58-a764-ba70d2197637","Type":"ContainerDied","Data":"3cd5625bc4788cfd69bbf7c66f183389c78a7bf63d0480aa8b3d44cec5f269e7"}
Sep 16 17:28:37.036658 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:37.036581    2577 scope.go:117] "RemoveContainer" containerID="3cd5625bc4788cfd69bbf7c66f183389c78a7bf63d0480aa8b3d44cec5f269e7"
Sep 16 17:28:38.041320 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:38.041283    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-jm27w" event={"ID":"146f8eb9-491d-4f58-a764-ba70d2197637","Type":"ContainerStarted","Data":"bb02c26b5bf3ce0b192e1b5d6bc2a552fb31837999c7e04843895aafc2894835"}
Sep 16 17:28:41.316563 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:41.316523    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:50.866094 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:50.866053    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e392d83-bd3a-4d0b-ac4f-300a6019c434-metrics-certs\") pod \"network-metrics-daemon-shvlb\" (UID: \"0e392d83-bd3a-4d0b-ac4f-300a6019c434\") " pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:28:50.868411 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:50.868390    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e392d83-bd3a-4d0b-ac4f-300a6019c434-metrics-certs\") pod \"network-metrics-daemon-shvlb\" (UID: \"0e392d83-bd3a-4d0b-ac4f-300a6019c434\") " pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:28:50.881380 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:50.881359    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hp4r4\""
Sep 16 17:28:50.889658 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:50.889643    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-shvlb"
Sep 16 17:28:51.157774 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:51.157725    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-shvlb"]
Sep 16 17:28:51.160322 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:28:51.160293    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e392d83_bd3a_4d0b_ac4f_300a6019c434.slice/crio-e9ff9a36e9970142d6c7162bdf78fd70b6a4c2e7cbd7dfba52ad569cd975c740 WatchSource:0}: Error finding container e9ff9a36e9970142d6c7162bdf78fd70b6a4c2e7cbd7dfba52ad569cd975c740: Status 404 returned error can't find the container with id e9ff9a36e9970142d6c7162bdf78fd70b6a4c2e7cbd7dfba52ad569cd975c740
Sep 16 17:28:51.413916 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:51.413809    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:51.498631 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:51.498597    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:52.091857 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:52.091771    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-shvlb" event={"ID":"0e392d83-bd3a-4d0b-ac4f-300a6019c434","Type":"ContainerStarted","Data":"ba8b2b6a4fccbde3b7d1bdd8c461f745682d3f16ee03e0d901db979bf0f80176"}
Sep 16 17:28:52.091857 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:52.091841    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-shvlb" event={"ID":"0e392d83-bd3a-4d0b-ac4f-300a6019c434","Type":"ContainerStarted","Data":"e9ff9a36e9970142d6c7162bdf78fd70b6a4c2e7cbd7dfba52ad569cd975c740"}
Sep 16 17:28:53.096456 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:53.096422    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-shvlb" event={"ID":"0e392d83-bd3a-4d0b-ac4f-300a6019c434","Type":"ContainerStarted","Data":"d0eb037cb9aafa0d3d0a145f0b23c69cb2b2681243a633909804a6bf2c814443"}
Sep 16 17:28:53.119486 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:53.119438    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-shvlb" podStartSLOduration=253.305796358 podStartE2EDuration="4m14.119423304s" totalImagesPullingTime="813.626946ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:28:53.118451999 +0000 UTC m=+254.481130796" watchObservedRunningTime="2026-09-16 17:28:53.119423304 +0000 UTC m=+254.482102089"
Sep 16 17:28:53.255737 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:53.255671    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:53.357514 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:53.357432    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:57.874879 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:57.874817    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:58.569067 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:58.569032    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:59.037329 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:59.037247    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:59.162703 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:59.162669    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:28:59.262515 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:28:59.262483    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:29:03.693246 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.693167    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-776b68d986-flcr6"]
Sep 16 17:29:03.693596 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.693423    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9d6a0b7-d55d-4919-b951-eb073d89c853" containerName="registry"
Sep 16 17:29:03.693596 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.693434    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9d6a0b7-d55d-4919-b951-eb073d89c853" containerName="registry"
Sep 16 17:29:03.693596 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.693479    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9d6a0b7-d55d-4919-b951-eb073d89c853" containerName="registry"
Sep 16 17:29:03.696443 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.696427    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-776b68d986-flcr6"
Sep 16 17:29:03.699472 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.699450    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 16 17:29:03.699735 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.699718    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 16 17:29:03.699827 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.699802    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 16 17:29:03.699953 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.699935    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-c66nr\""
Sep 16 17:29:03.700252 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.700237    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 16 17:29:03.700306 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.700240    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 16 17:29:03.716090 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.716057    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-776b68d986-flcr6"]
Sep 16 17:29:03.716617 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.716599    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 16 17:29:03.862000 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.861949    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73d2b32c-116f-4692-bf1e-f75beae96d63-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-776b68d986-flcr6\" (UID: \"73d2b32c-116f-4692-bf1e-f75beae96d63\") " pod="openshift-monitoring/telemeter-client-776b68d986-flcr6"
Sep 16 17:29:03.862000 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.862002    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/73d2b32c-116f-4692-bf1e-f75beae96d63-federate-client-tls\") pod \"telemeter-client-776b68d986-flcr6\" (UID: \"73d2b32c-116f-4692-bf1e-f75beae96d63\") " pod="openshift-monitoring/telemeter-client-776b68d986-flcr6"
Sep 16 17:29:03.862267 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.862025    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fq8h\" (UniqueName: \"kubernetes.io/projected/73d2b32c-116f-4692-bf1e-f75beae96d63-kube-api-access-5fq8h\") pod \"telemeter-client-776b68d986-flcr6\" (UID: \"73d2b32c-116f-4692-bf1e-f75beae96d63\") " pod="openshift-monitoring/telemeter-client-776b68d986-flcr6"
Sep 16 17:29:03.862267 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.862076    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/73d2b32c-116f-4692-bf1e-f75beae96d63-secret-telemeter-client\") pod \"telemeter-client-776b68d986-flcr6\" (UID: \"73d2b32c-116f-4692-bf1e-f75beae96d63\") " pod="openshift-monitoring/telemeter-client-776b68d986-flcr6"
Sep 16 17:29:03.862267 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.862097    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73d2b32c-116f-4692-bf1e-f75beae96d63-telemeter-trusted-ca-bundle\") pod \"telemeter-client-776b68d986-flcr6\" (UID: \"73d2b32c-116f-4692-bf1e-f75beae96d63\") " pod="openshift-monitoring/telemeter-client-776b68d986-flcr6"
Sep 16 17:29:03.862267 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.862129    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73d2b32c-116f-4692-bf1e-f75beae96d63-serving-certs-ca-bundle\") pod \"telemeter-client-776b68d986-flcr6\" (UID: \"73d2b32c-116f-4692-bf1e-f75beae96d63\") " pod="openshift-monitoring/telemeter-client-776b68d986-flcr6"
Sep 16 17:29:03.862267 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.862152    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/73d2b32c-116f-4692-bf1e-f75beae96d63-telemeter-client-tls\") pod \"telemeter-client-776b68d986-flcr6\" (UID: \"73d2b32c-116f-4692-bf1e-f75beae96d63\") " pod="openshift-monitoring/telemeter-client-776b68d986-flcr6"
Sep 16 17:29:03.862267 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.862178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73d2b32c-116f-4692-bf1e-f75beae96d63-metrics-client-ca\") pod \"telemeter-client-776b68d986-flcr6\" (UID: \"73d2b32c-116f-4692-bf1e-f75beae96d63\") " pod="openshift-monitoring/telemeter-client-776b68d986-flcr6"
Sep 16 17:29:03.963357 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.963272    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73d2b32c-116f-4692-bf1e-f75beae96d63-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-776b68d986-flcr6\" (UID: \"73d2b32c-116f-4692-bf1e-f75beae96d63\") " pod="openshift-monitoring/telemeter-client-776b68d986-flcr6"
Sep 16 17:29:03.963357 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.963314    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/73d2b32c-116f-4692-bf1e-f75beae96d63-federate-client-tls\") pod \"telemeter-client-776b68d986-flcr6\" (UID: \"73d2b32c-116f-4692-bf1e-f75beae96d63\") " pod="openshift-monitoring/telemeter-client-776b68d986-flcr6"
Sep 16 17:29:03.963357 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.963331    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fq8h\" (UniqueName: \"kubernetes.io/projected/73d2b32c-116f-4692-bf1e-f75beae96d63-kube-api-access-5fq8h\") pod \"telemeter-client-776b68d986-flcr6\" (UID: \"73d2b32c-116f-4692-bf1e-f75beae96d63\") " pod="openshift-monitoring/telemeter-client-776b68d986-flcr6"
Sep 16 17:29:03.963605 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.963363    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/73d2b32c-116f-4692-bf1e-f75beae96d63-secret-telemeter-client\") pod \"telemeter-client-776b68d986-flcr6\" (UID: \"73d2b32c-116f-4692-bf1e-f75beae96d63\") " pod="openshift-monitoring/telemeter-client-776b68d986-flcr6"
Sep 16 17:29:03.963605 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.963383    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73d2b32c-116f-4692-bf1e-f75beae96d63-telemeter-trusted-ca-bundle\") pod \"telemeter-client-776b68d986-flcr6\" (UID: \"73d2b32c-116f-4692-bf1e-f75beae96d63\") " pod="openshift-monitoring/telemeter-client-776b68d986-flcr6"
Sep 16 17:29:03.963605 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.963402    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73d2b32c-116f-4692-bf1e-f75beae96d63-serving-certs-ca-bundle\") pod \"telemeter-client-776b68d986-flcr6\" (UID: \"73d2b32c-116f-4692-bf1e-f75beae96d63\") " pod="openshift-monitoring/telemeter-client-776b68d986-flcr6"
Sep 16 17:29:03.963605 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.963423    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/73d2b32c-116f-4692-bf1e-f75beae96d63-telemeter-client-tls\") pod \"telemeter-client-776b68d986-flcr6\" (UID: \"73d2b32c-116f-4692-bf1e-f75beae96d63\") " pod="openshift-monitoring/telemeter-client-776b68d986-flcr6"
Sep 16 17:29:03.963605 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.963447    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73d2b32c-116f-4692-bf1e-f75beae96d63-metrics-client-ca\") pod \"telemeter-client-776b68d986-flcr6\" (UID: \"73d2b32c-116f-4692-bf1e-f75beae96d63\") " pod="openshift-monitoring/telemeter-client-776b68d986-flcr6"
Sep 16 17:29:03.964292 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.964263    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73d2b32c-116f-4692-bf1e-f75beae96d63-metrics-client-ca\") pod \"telemeter-client-776b68d986-flcr6\" (UID: \"73d2b32c-116f-4692-bf1e-f75beae96d63\") " pod="openshift-monitoring/telemeter-client-776b68d986-flcr6"
Sep 16 17:29:03.964410 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.964263    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73d2b32c-116f-4692-bf1e-f75beae96d63-serving-certs-ca-bundle\") pod \"telemeter-client-776b68d986-flcr6\" (UID: \"73d2b32c-116f-4692-bf1e-f75beae96d63\") " pod="openshift-monitoring/telemeter-client-776b68d986-flcr6"
Sep 16 17:29:03.964484 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.964465    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73d2b32c-116f-4692-bf1e-f75beae96d63-telemeter-trusted-ca-bundle\") pod \"telemeter-client-776b68d986-flcr6\" (UID: \"73d2b32c-116f-4692-bf1e-f75beae96d63\") " pod="openshift-monitoring/telemeter-client-776b68d986-flcr6"
Sep 16 17:29:03.966082 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.966057    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/73d2b32c-116f-4692-bf1e-f75beae96d63-secret-telemeter-client\") pod \"telemeter-client-776b68d986-flcr6\" (UID: \"73d2b32c-116f-4692-bf1e-f75beae96d63\") " pod="openshift-monitoring/telemeter-client-776b68d986-flcr6"
Sep 16 17:29:03.966082 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.966075    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/73d2b32c-116f-4692-bf1e-f75beae96d63-telemeter-client-tls\") pod \"telemeter-client-776b68d986-flcr6\" (UID: \"73d2b32c-116f-4692-bf1e-f75beae96d63\") " pod="openshift-monitoring/telemeter-client-776b68d986-flcr6"
Sep 16 17:29:03.966239 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.966227    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/73d2b32c-116f-4692-bf1e-f75beae96d63-federate-client-tls\") pod \"telemeter-client-776b68d986-flcr6\" (UID: \"73d2b32c-116f-4692-bf1e-f75beae96d63\") " pod="openshift-monitoring/telemeter-client-776b68d986-flcr6"
Sep 16 17:29:03.966295 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.966248    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73d2b32c-116f-4692-bf1e-f75beae96d63-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-776b68d986-flcr6\" (UID: \"73d2b32c-116f-4692-bf1e-f75beae96d63\") " pod="openshift-monitoring/telemeter-client-776b68d986-flcr6"
Sep 16 17:29:03.983170 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:03.983150    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fq8h\" (UniqueName: \"kubernetes.io/projected/73d2b32c-116f-4692-bf1e-f75beae96d63-kube-api-access-5fq8h\") pod \"telemeter-client-776b68d986-flcr6\" (UID: \"73d2b32c-116f-4692-bf1e-f75beae96d63\") " pod="openshift-monitoring/telemeter-client-776b68d986-flcr6"
Sep 16 17:29:04.011122 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:04.011097    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-776b68d986-flcr6"
Sep 16 17:29:04.293512 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:04.293441    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-776b68d986-flcr6"]
Sep 16 17:29:04.296240 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:29:04.296215    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73d2b32c_116f_4692_bf1e_f75beae96d63.slice/crio-90fe82395a1e992018752c2c3290a9cdee45c3f72d6b62b3c7e449662a0e1880 WatchSource:0}: Error finding container 90fe82395a1e992018752c2c3290a9cdee45c3f72d6b62b3c7e449662a0e1880: Status 404 returned error can't find the container with id 90fe82395a1e992018752c2c3290a9cdee45c3f72d6b62b3c7e449662a0e1880
Sep 16 17:29:05.136639 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:05.136606    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-776b68d986-flcr6" event={"ID":"73d2b32c-116f-4692-bf1e-f75beae96d63","Type":"ContainerStarted","Data":"90fe82395a1e992018752c2c3290a9cdee45c3f72d6b62b3c7e449662a0e1880"}
Sep 16 17:29:07.146322 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:07.146282    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-776b68d986-flcr6" event={"ID":"73d2b32c-116f-4692-bf1e-f75beae96d63","Type":"ContainerStarted","Data":"c83aa45fdadc0cd3b4b2cc3934a2453c993552e109125896c76f9437868b22d9"}
Sep 16 17:29:08.152425 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:08.152390    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-776b68d986-flcr6" event={"ID":"73d2b32c-116f-4692-bf1e-f75beae96d63","Type":"ContainerStarted","Data":"a3f1437d21e23cddc7eff2d8039d5248bbe96ac66f49e3e90a867810335f05ac"}
Sep 16 17:29:08.152788 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:08.152446    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-776b68d986-flcr6" event={"ID":"73d2b32c-116f-4692-bf1e-f75beae96d63","Type":"ContainerStarted","Data":"35a5e0814175996c81b726ac159ab8de69ceaa44479e87104f1beab463d3e8b2"}
Sep 16 17:29:08.204277 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:08.204233    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-776b68d986-flcr6" podStartSLOduration=2.224178189 podStartE2EDuration="5.204217969s" totalImagesPullingTime="2.98003978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:29:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:29:08.199678576 +0000 UTC m=+269.562357459" watchObservedRunningTime="2026-09-16 17:29:08.204217969 +0000 UTC m=+269.566896755"
Sep 16 17:29:08.426493 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:08.426404    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:29:08.529235 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:08.529204    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:29:19.246840 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:19.246807    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:29:19.346919 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:19.346885    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:29:19.417533 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:19.417506    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:29:22.594239 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:22.594205    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:29:22.661030 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:22.661000    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:29:38.996858 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:38.996830    2577 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 17:29:52.805706 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:52.805669    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:29:52.858560 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:52.858519    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:29:53.418051 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:29:53.418008    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:30:13.857431 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:30:13.857395    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:30:13.913315 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:30:13.913275    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:30:16.408302 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:30:16.408271    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:30:16.509165 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:30:16.509129    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:30:20.534196 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:30:20.534157    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:30:25.827297 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:30:25.827234    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:30:25.998907 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:30:25.998854    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:30:26.481259 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:30:26.481224    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:30:26.582915 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:30:26.582853    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:30:26.643800 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:30:26.643761    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:30:37.288556 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:30:37.288466    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:30:37.389139 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:30:37.389108    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:30:37.455596 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:30:37.455552    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:32:36.320679 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:32:36.320644    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:32:37.810139 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:32:37.810103    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:32:44.488488 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:32:44.488447    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:32:45.076259 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:32:45.076214    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:32:45.192603 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:32:45.192570    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:32:45.283470 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:32:45.283435    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:32:46.732406 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:32:46.732372    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a46fjg"]
Sep 16 17:32:46.735748 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:32:46.735730    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a46fjg"
Sep 16 17:32:46.738490 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:32:46.738474    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-8lpxq\""
Sep 16 17:32:46.738691 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:32:46.738674    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 17:32:46.739712 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:32:46.739696    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 17:32:46.748122 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:32:46.748096    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a46fjg"]
Sep 16 17:32:46.804243 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:32:46.804204    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f245ad79-f7fa-4b70-aac7-fafef4f89e0a-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a46fjg\" (UID: \"f245ad79-f7fa-4b70-aac7-fafef4f89e0a\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a46fjg"
Sep 16 17:32:46.804402 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:32:46.804286    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrmlg\" (UniqueName: \"kubernetes.io/projected/f245ad79-f7fa-4b70-aac7-fafef4f89e0a-kube-api-access-rrmlg\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a46fjg\" (UID: \"f245ad79-f7fa-4b70-aac7-fafef4f89e0a\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a46fjg"
Sep 16 17:32:46.804402 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:32:46.804339    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f245ad79-f7fa-4b70-aac7-fafef4f89e0a-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a46fjg\" (UID: \"f245ad79-f7fa-4b70-aac7-fafef4f89e0a\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a46fjg"
Sep 16 17:32:46.905036 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:32:46.905001    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrmlg\" (UniqueName: \"kubernetes.io/projected/f245ad79-f7fa-4b70-aac7-fafef4f89e0a-kube-api-access-rrmlg\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a46fjg\" (UID: \"f245ad79-f7fa-4b70-aac7-fafef4f89e0a\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a46fjg"
Sep 16 17:32:46.905036 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:32:46.905040    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f245ad79-f7fa-4b70-aac7-fafef4f89e0a-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a46fjg\" (UID: \"f245ad79-f7fa-4b70-aac7-fafef4f89e0a\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a46fjg"
Sep 16 17:32:46.905215 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:32:46.905095    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f245ad79-f7fa-4b70-aac7-fafef4f89e0a-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a46fjg\" (UID: \"f245ad79-f7fa-4b70-aac7-fafef4f89e0a\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a46fjg"
Sep 16 17:32:46.905417 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:32:46.905399    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f245ad79-f7fa-4b70-aac7-fafef4f89e0a-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a46fjg\" (UID: \"f245ad79-f7fa-4b70-aac7-fafef4f89e0a\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a46fjg"
Sep 16 17:32:46.905453 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:32:46.905437    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f245ad79-f7fa-4b70-aac7-fafef4f89e0a-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a46fjg\" (UID: \"f245ad79-f7fa-4b70-aac7-fafef4f89e0a\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a46fjg"
Sep 16 17:32:46.926250 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:32:46.926219    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrmlg\" (UniqueName: \"kubernetes.io/projected/f245ad79-f7fa-4b70-aac7-fafef4f89e0a-kube-api-access-rrmlg\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a46fjg\" (UID: \"f245ad79-f7fa-4b70-aac7-fafef4f89e0a\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a46fjg"
Sep 16 17:32:47.044211 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:32:47.044130    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a46fjg"
Sep 16 17:32:47.156310 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:32:47.156281    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:32:47.287194 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:32:47.287163    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:32:47.337292 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:32:47.337265    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a46fjg"]
Sep 16 17:32:47.340291 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:32:47.340261    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf245ad79_f7fa_4b70_aac7_fafef4f89e0a.slice/crio-44f060bc694ff2768e652d30b75a4c8dfc6fd6fca03a5b3175529f3cad84dd3c WatchSource:0}: Error finding container 44f060bc694ff2768e652d30b75a4c8dfc6fd6fca03a5b3175529f3cad84dd3c: Status 404 returned error can't find the container with id 44f060bc694ff2768e652d30b75a4c8dfc6fd6fca03a5b3175529f3cad84dd3c
Sep 16 17:32:47.342282 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:32:47.342265    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 17:32:47.419430 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:32:47.419403    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:32:47.910545 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:32:47.910511    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a46fjg" event={"ID":"f245ad79-f7fa-4b70-aac7-fafef4f89e0a","Type":"ContainerStarted","Data":"44f060bc694ff2768e652d30b75a4c8dfc6fd6fca03a5b3175529f3cad84dd3c"}
Sep 16 17:32:55.941376 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:32:55.941336    2577 generic.go:356] "Generic (PLEG): container finished" podID="f245ad79-f7fa-4b70-aac7-fafef4f89e0a" containerID="2c6398b37aeb1e754e7e3b934e00bbd12f3fcd3477d381a370d15bb14263cd56" exitCode=0
Sep 16 17:32:55.941835 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:32:55.941426    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a46fjg" event={"ID":"f245ad79-f7fa-4b70-aac7-fafef4f89e0a","Type":"ContainerDied","Data":"2c6398b37aeb1e754e7e3b934e00bbd12f3fcd3477d381a370d15bb14263cd56"}
Sep 16 17:32:57.951723 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:32:57.951688    2577 generic.go:356] "Generic (PLEG): container finished" podID="f245ad79-f7fa-4b70-aac7-fafef4f89e0a" containerID="5c436f459f3986f5133e8d3feb03baaae6b485da1c1ad035e17a0529dab8f4e3" exitCode=0
Sep 16 17:32:57.951723 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:32:57.951724    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a46fjg" event={"ID":"f245ad79-f7fa-4b70-aac7-fafef4f89e0a","Type":"ContainerDied","Data":"5c436f459f3986f5133e8d3feb03baaae6b485da1c1ad035e17a0529dab8f4e3"}
Sep 16 17:33:03.977440 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:03.977398    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a46fjg" event={"ID":"f245ad79-f7fa-4b70-aac7-fafef4f89e0a","Type":"ContainerStarted","Data":"8be30d949d6ac3ed2a5ce138a8b11226a6bf097e4327fda67df6da643af5fa69"}
Sep 16 17:33:03.997448 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:03.997396    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a46fjg" podStartSLOduration=2.379348852 podStartE2EDuration="17.997381919s" totalImagesPullingTime="15.618033067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:32:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:33:03.995952406 +0000 UTC m=+505.358631192" watchObservedRunningTime="2026-09-16 17:33:03.997381919 +0000 UTC m=+505.360060705"
Sep 16 17:33:04.984041 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:04.984008    2577 generic.go:356] "Generic (PLEG): container finished" podID="f245ad79-f7fa-4b70-aac7-fafef4f89e0a" containerID="8be30d949d6ac3ed2a5ce138a8b11226a6bf097e4327fda67df6da643af5fa69" exitCode=0
Sep 16 17:33:04.984420 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:04.984097    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a46fjg" event={"ID":"f245ad79-f7fa-4b70-aac7-fafef4f89e0a","Type":"ContainerDied","Data":"8be30d949d6ac3ed2a5ce138a8b11226a6bf097e4327fda67df6da643af5fa69"}
Sep 16 17:33:06.103533 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:06.103505    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a46fjg"
Sep 16 17:33:06.163280 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:06.163249    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f245ad79-f7fa-4b70-aac7-fafef4f89e0a-util\") pod \"f245ad79-f7fa-4b70-aac7-fafef4f89e0a\" (UID: \"f245ad79-f7fa-4b70-aac7-fafef4f89e0a\") "
Sep 16 17:33:06.163280 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:06.163282    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrmlg\" (UniqueName: \"kubernetes.io/projected/f245ad79-f7fa-4b70-aac7-fafef4f89e0a-kube-api-access-rrmlg\") pod \"f245ad79-f7fa-4b70-aac7-fafef4f89e0a\" (UID: \"f245ad79-f7fa-4b70-aac7-fafef4f89e0a\") "
Sep 16 17:33:06.163488 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:06.163309    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f245ad79-f7fa-4b70-aac7-fafef4f89e0a-bundle\") pod \"f245ad79-f7fa-4b70-aac7-fafef4f89e0a\" (UID: \"f245ad79-f7fa-4b70-aac7-fafef4f89e0a\") "
Sep 16 17:33:06.163997 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:06.163969    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f245ad79-f7fa-4b70-aac7-fafef4f89e0a-bundle" (OuterVolumeSpecName: "bundle") pod "f245ad79-f7fa-4b70-aac7-fafef4f89e0a" (UID: "f245ad79-f7fa-4b70-aac7-fafef4f89e0a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:33:06.165423 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:06.165403    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f245ad79-f7fa-4b70-aac7-fafef4f89e0a-kube-api-access-rrmlg" (OuterVolumeSpecName: "kube-api-access-rrmlg") pod "f245ad79-f7fa-4b70-aac7-fafef4f89e0a" (UID: "f245ad79-f7fa-4b70-aac7-fafef4f89e0a"). InnerVolumeSpecName "kube-api-access-rrmlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:33:06.167830 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:06.167805    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f245ad79-f7fa-4b70-aac7-fafef4f89e0a-util" (OuterVolumeSpecName: "util") pod "f245ad79-f7fa-4b70-aac7-fafef4f89e0a" (UID: "f245ad79-f7fa-4b70-aac7-fafef4f89e0a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:33:06.264045 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:06.263969    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f245ad79-f7fa-4b70-aac7-fafef4f89e0a-util\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:33:06.264045 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:06.263994    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrmlg\" (UniqueName: \"kubernetes.io/projected/f245ad79-f7fa-4b70-aac7-fafef4f89e0a-kube-api-access-rrmlg\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:33:06.264045 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:06.264003    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f245ad79-f7fa-4b70-aac7-fafef4f89e0a-bundle\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:33:06.992780 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:06.992755    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a46fjg"
Sep 16 17:33:06.992780 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:06.992760    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a46fjg" event={"ID":"f245ad79-f7fa-4b70-aac7-fafef4f89e0a","Type":"ContainerDied","Data":"44f060bc694ff2768e652d30b75a4c8dfc6fd6fca03a5b3175529f3cad84dd3c"}
Sep 16 17:33:06.993011 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:06.992803    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44f060bc694ff2768e652d30b75a4c8dfc6fd6fca03a5b3175529f3cad84dd3c"
Sep 16 17:33:08.100307 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:08.100276    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:08.223415 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:08.223382    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:09.807880 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:09.807832    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-p7vpt"]
Sep 16 17:33:09.808263 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:09.808103    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f245ad79-f7fa-4b70-aac7-fafef4f89e0a" containerName="util"
Sep 16 17:33:09.808263 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:09.808114    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f245ad79-f7fa-4b70-aac7-fafef4f89e0a" containerName="util"
Sep 16 17:33:09.808263 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:09.808126    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f245ad79-f7fa-4b70-aac7-fafef4f89e0a" containerName="pull"
Sep 16 17:33:09.808263 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:09.808132    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f245ad79-f7fa-4b70-aac7-fafef4f89e0a" containerName="pull"
Sep 16 17:33:09.808263 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:09.808138    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f245ad79-f7fa-4b70-aac7-fafef4f89e0a" containerName="extract"
Sep 16 17:33:09.808263 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:09.808143    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f245ad79-f7fa-4b70-aac7-fafef4f89e0a" containerName="extract"
Sep 16 17:33:09.808263 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:09.808197    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f245ad79-f7fa-4b70-aac7-fafef4f89e0a" containerName="extract"
Sep 16 17:33:09.844849 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:09.844819    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-p7vpt"
Sep 16 17:33:09.846652 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:09.846625    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-p7vpt"]
Sep 16 17:33:09.849057 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:09.849028    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 16 17:33:09.849199 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:09.849173    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 16 17:33:09.849351 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:09.849335    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-gftp8\""
Sep 16 17:33:09.854975 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:09.854954    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 16 17:33:09.993093 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:09.993057    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:09.993333 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:09.993267    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr87k\" (UniqueName: \"kubernetes.io/projected/24b17a32-616d-40a7-b156-a466c820c536-kube-api-access-fr87k\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-p7vpt\" (UID: \"24b17a32-616d-40a7-b156-a466c820c536\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-p7vpt"
Sep 16 17:33:09.993333 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:09.993306    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/24b17a32-616d-40a7-b156-a466c820c536-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-p7vpt\" (UID: \"24b17a32-616d-40a7-b156-a466c820c536\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-p7vpt"
Sep 16 17:33:10.093640 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:10.093609    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr87k\" (UniqueName: \"kubernetes.io/projected/24b17a32-616d-40a7-b156-a466c820c536-kube-api-access-fr87k\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-p7vpt\" (UID: \"24b17a32-616d-40a7-b156-a466c820c536\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-p7vpt"
Sep 16 17:33:10.093826 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:10.093645    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/24b17a32-616d-40a7-b156-a466c820c536-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-p7vpt\" (UID: \"24b17a32-616d-40a7-b156-a466c820c536\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-p7vpt"
Sep 16 17:33:10.096017 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:10.095993    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/24b17a32-616d-40a7-b156-a466c820c536-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-p7vpt\" (UID: \"24b17a32-616d-40a7-b156-a466c820c536\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-p7vpt"
Sep 16 17:33:10.105620 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:10.105587    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr87k\" (UniqueName: \"kubernetes.io/projected/24b17a32-616d-40a7-b156-a466c820c536-kube-api-access-fr87k\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-p7vpt\" (UID: \"24b17a32-616d-40a7-b156-a466c820c536\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-p7vpt"
Sep 16 17:33:10.108655 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:10.108629    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:10.155625 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:10.155598    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-p7vpt"
Sep 16 17:33:10.458260 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:10.458187    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-p7vpt"]
Sep 16 17:33:10.461454 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:33:10.461429    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b17a32_616d_40a7_b156_a466c820c536.slice/crio-43a07c74ddeb1c5f19aae29207389acdb53af5ef853dc00a7e839f89a195aba2 WatchSource:0}: Error finding container 43a07c74ddeb1c5f19aae29207389acdb53af5ef853dc00a7e839f89a195aba2: Status 404 returned error can't find the container with id 43a07c74ddeb1c5f19aae29207389acdb53af5ef853dc00a7e839f89a195aba2
Sep 16 17:33:11.013958 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:11.013915    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-p7vpt" event={"ID":"24b17a32-616d-40a7-b156-a466c820c536","Type":"ContainerStarted","Data":"43a07c74ddeb1c5f19aae29207389acdb53af5ef853dc00a7e839f89a195aba2"}
Sep 16 17:33:14.412857 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:14.412820    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:14.761744 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:14.761663    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:14.772147 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:14.772119    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-v68f2"]
Sep 16 17:33:14.774249 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:14.774235    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-v68f2"
Sep 16 17:33:14.778112 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:14.778089    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 16 17:33:14.778112 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:14.778104    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-q4dsj\""
Sep 16 17:33:14.778279 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:14.778124    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 16 17:33:14.792376 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:14.792351    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-v68f2"]
Sep 16 17:33:14.869969 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:14.869931    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:14.933048 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:14.933013    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppqvk\" (UniqueName: \"kubernetes.io/projected/1d7f4c40-b184-4988-9c18-27ab2c8ea02b-kube-api-access-ppqvk\") pod \"keda-operator-579c8958f7-v68f2\" (UID: \"1d7f4c40-b184-4988-9c18-27ab2c8ea02b\") " pod="openshift-keda/keda-operator-579c8958f7-v68f2"
Sep 16 17:33:14.933210 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:14.933069    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1d7f4c40-b184-4988-9c18-27ab2c8ea02b-certificates\") pod \"keda-operator-579c8958f7-v68f2\" (UID: \"1d7f4c40-b184-4988-9c18-27ab2c8ea02b\") " pod="openshift-keda/keda-operator-579c8958f7-v68f2"
Sep 16 17:33:14.933210 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:14.933123    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1d7f4c40-b184-4988-9c18-27ab2c8ea02b-cabundle0\") pod \"keda-operator-579c8958f7-v68f2\" (UID: \"1d7f4c40-b184-4988-9c18-27ab2c8ea02b\") " pod="openshift-keda/keda-operator-579c8958f7-v68f2"
Sep 16 17:33:14.965750 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:14.965712    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:14.985338 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:14.985302    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-6fpjp"]
Sep 16 17:33:14.987533 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:14.987512    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-6fpjp"
Sep 16 17:33:14.990633 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:14.990608    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 16 17:33:15.001121 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:15.001098    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-6fpjp"]
Sep 16 17:33:15.033985 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:15.033903    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppqvk\" (UniqueName: \"kubernetes.io/projected/1d7f4c40-b184-4988-9c18-27ab2c8ea02b-kube-api-access-ppqvk\") pod \"keda-operator-579c8958f7-v68f2\" (UID: \"1d7f4c40-b184-4988-9c18-27ab2c8ea02b\") " pod="openshift-keda/keda-operator-579c8958f7-v68f2"
Sep 16 17:33:15.033985 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:15.033972    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1d7f4c40-b184-4988-9c18-27ab2c8ea02b-certificates\") pod \"keda-operator-579c8958f7-v68f2\" (UID: \"1d7f4c40-b184-4988-9c18-27ab2c8ea02b\") " pod="openshift-keda/keda-operator-579c8958f7-v68f2"
Sep 16 17:33:15.034167 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:15.034008    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1d7f4c40-b184-4988-9c18-27ab2c8ea02b-cabundle0\") pod \"keda-operator-579c8958f7-v68f2\" (UID: \"1d7f4c40-b184-4988-9c18-27ab2c8ea02b\") " pod="openshift-keda/keda-operator-579c8958f7-v68f2"
Sep 16 17:33:15.034167 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:33:15.034108    2577 secret.go:281] references non-existent secret key: ca.crt
Sep 16 17:33:15.034167 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:33:15.034127    2577 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 16 17:33:15.034167 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:33:15.034135    2577 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-v68f2: references non-existent secret key: ca.crt
Sep 16 17:33:15.034295 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:33:15.034197    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d7f4c40-b184-4988-9c18-27ab2c8ea02b-certificates podName:1d7f4c40-b184-4988-9c18-27ab2c8ea02b nodeName:}" failed. No retries permitted until 2026-09-16 17:33:15.534180138 +0000 UTC m=+516.896858907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/1d7f4c40-b184-4988-9c18-27ab2c8ea02b-certificates") pod "keda-operator-579c8958f7-v68f2" (UID: "1d7f4c40-b184-4988-9c18-27ab2c8ea02b") : references non-existent secret key: ca.crt
Sep 16 17:33:15.034823 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:15.034797    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1d7f4c40-b184-4988-9c18-27ab2c8ea02b-cabundle0\") pod \"keda-operator-579c8958f7-v68f2\" (UID: \"1d7f4c40-b184-4988-9c18-27ab2c8ea02b\") " pod="openshift-keda/keda-operator-579c8958f7-v68f2"
Sep 16 17:33:15.038698 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:15.038671    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-p7vpt" event={"ID":"24b17a32-616d-40a7-b156-a466c820c536","Type":"ContainerStarted","Data":"ed73c1d1cd1d721c3f14e88324e9d9a0869df1455eba85a6af11c12a0c349e7b"}
Sep 16 17:33:15.038829 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:15.038811    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-p7vpt"
Sep 16 17:33:15.049114 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:15.049095    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppqvk\" (UniqueName: \"kubernetes.io/projected/1d7f4c40-b184-4988-9c18-27ab2c8ea02b-kube-api-access-ppqvk\") pod \"keda-operator-579c8958f7-v68f2\" (UID: \"1d7f4c40-b184-4988-9c18-27ab2c8ea02b\") " pod="openshift-keda/keda-operator-579c8958f7-v68f2"
Sep 16 17:33:15.067309 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:15.067257    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-p7vpt" podStartSLOduration=2.559115298 podStartE2EDuration="6.067241704s" totalImagesPullingTime="3.508126406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:33:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:33:15.0658578 +0000 UTC m=+516.428536586" watchObservedRunningTime="2026-09-16 17:33:15.067241704 +0000 UTC m=+516.429920490"
Sep 16 17:33:15.110583 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:15.110548    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:15.134631 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:15.134602    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/5c1d21e6-c8e9-4321-89b0-a6246e89db0d-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-6fpjp\" (UID: \"5c1d21e6-c8e9-4321-89b0-a6246e89db0d\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-6fpjp"
Sep 16 17:33:15.134776 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:15.134651    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgcgs\" (UniqueName: \"kubernetes.io/projected/5c1d21e6-c8e9-4321-89b0-a6246e89db0d-kube-api-access-pgcgs\") pod \"keda-metrics-apiserver-77857b7b98-6fpjp\" (UID: \"5c1d21e6-c8e9-4321-89b0-a6246e89db0d\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-6fpjp"
Sep 16 17:33:15.134776 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:15.134738    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5c1d21e6-c8e9-4321-89b0-a6246e89db0d-certificates\") pod \"keda-metrics-apiserver-77857b7b98-6fpjp\" (UID: \"5c1d21e6-c8e9-4321-89b0-a6246e89db0d\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-6fpjp"
Sep 16 17:33:15.235152 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:15.235119    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/5c1d21e6-c8e9-4321-89b0-a6246e89db0d-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-6fpjp\" (UID: \"5c1d21e6-c8e9-4321-89b0-a6246e89db0d\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-6fpjp"
Sep 16 17:33:15.235292 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:15.235159    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgcgs\" (UniqueName: \"kubernetes.io/projected/5c1d21e6-c8e9-4321-89b0-a6246e89db0d-kube-api-access-pgcgs\") pod \"keda-metrics-apiserver-77857b7b98-6fpjp\" (UID: \"5c1d21e6-c8e9-4321-89b0-a6246e89db0d\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-6fpjp"
Sep 16 17:33:15.235292 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:15.235202    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5c1d21e6-c8e9-4321-89b0-a6246e89db0d-certificates\") pod \"keda-metrics-apiserver-77857b7b98-6fpjp\" (UID: \"5c1d21e6-c8e9-4321-89b0-a6246e89db0d\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-6fpjp"
Sep 16 17:33:15.235359 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:33:15.235313    2577 secret.go:281] references non-existent secret key: tls.crt
Sep 16 17:33:15.235359 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:33:15.235329    2577 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 16 17:33:15.235359 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:33:15.235351    2577 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-6fpjp: references non-existent secret key: tls.crt
Sep 16 17:33:15.235514 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:33:15.235407    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5c1d21e6-c8e9-4321-89b0-a6246e89db0d-certificates podName:5c1d21e6-c8e9-4321-89b0-a6246e89db0d nodeName:}" failed. No retries permitted until 2026-09-16 17:33:15.73539165 +0000 UTC m=+517.098070417 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/5c1d21e6-c8e9-4321-89b0-a6246e89db0d-certificates") pod "keda-metrics-apiserver-77857b7b98-6fpjp" (UID: "5c1d21e6-c8e9-4321-89b0-a6246e89db0d") : references non-existent secret key: tls.crt
Sep 16 17:33:15.235514 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:15.235489    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/5c1d21e6-c8e9-4321-89b0-a6246e89db0d-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-6fpjp\" (UID: \"5c1d21e6-c8e9-4321-89b0-a6246e89db0d\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-6fpjp"
Sep 16 17:33:15.243561 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:15.243536    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgcgs\" (UniqueName: \"kubernetes.io/projected/5c1d21e6-c8e9-4321-89b0-a6246e89db0d-kube-api-access-pgcgs\") pod \"keda-metrics-apiserver-77857b7b98-6fpjp\" (UID: \"5c1d21e6-c8e9-4321-89b0-a6246e89db0d\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-6fpjp"
Sep 16 17:33:15.538228 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:15.538193    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1d7f4c40-b184-4988-9c18-27ab2c8ea02b-certificates\") pod \"keda-operator-579c8958f7-v68f2\" (UID: \"1d7f4c40-b184-4988-9c18-27ab2c8ea02b\") " pod="openshift-keda/keda-operator-579c8958f7-v68f2"
Sep 16 17:33:15.540554 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:15.540525    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1d7f4c40-b184-4988-9c18-27ab2c8ea02b-certificates\") pod \"keda-operator-579c8958f7-v68f2\" (UID: \"1d7f4c40-b184-4988-9c18-27ab2c8ea02b\") " pod="openshift-keda/keda-operator-579c8958f7-v68f2"
Sep 16 17:33:15.586273 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:15.586236    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:15.683912 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:15.683856    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-v68f2"
Sep 16 17:33:15.687887 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:15.687846    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:15.740672 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:15.740639    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5c1d21e6-c8e9-4321-89b0-a6246e89db0d-certificates\") pod \"keda-metrics-apiserver-77857b7b98-6fpjp\" (UID: \"5c1d21e6-c8e9-4321-89b0-a6246e89db0d\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-6fpjp"
Sep 16 17:33:15.743966 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:15.743908    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5c1d21e6-c8e9-4321-89b0-a6246e89db0d-certificates\") pod \"keda-metrics-apiserver-77857b7b98-6fpjp\" (UID: \"5c1d21e6-c8e9-4321-89b0-a6246e89db0d\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-6fpjp"
Sep 16 17:33:15.897552 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:15.897518    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-6fpjp"
Sep 16 17:33:15.957544 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:15.957489    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-v68f2"]
Sep 16 17:33:15.960036 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:33:15.959809    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7f4c40_b184_4988_9c18_27ab2c8ea02b.slice/crio-972076c4fc3643f779e48ff5da36c0208092250272ab34f9f77ad39c266b1aaa WatchSource:0}: Error finding container 972076c4fc3643f779e48ff5da36c0208092250272ab34f9f77ad39c266b1aaa: Status 404 returned error can't find the container with id 972076c4fc3643f779e48ff5da36c0208092250272ab34f9f77ad39c266b1aaa
Sep 16 17:33:16.046931 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:16.046893    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-v68f2" event={"ID":"1d7f4c40-b184-4988-9c18-27ab2c8ea02b","Type":"ContainerStarted","Data":"972076c4fc3643f779e48ff5da36c0208092250272ab34f9f77ad39c266b1aaa"}
Sep 16 17:33:16.056226 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:16.056185    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:16.161176 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:16.161096    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:16.176392 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:16.176314    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-6fpjp"]
Sep 16 17:33:16.179064 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:33:16.179038    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c1d21e6_c8e9_4321_89b0_a6246e89db0d.slice/crio-fcb93d278b0e15550319f205617fe59147f6a7a955d72282ae73be01af8e7735 WatchSource:0}: Error finding container fcb93d278b0e15550319f205617fe59147f6a7a955d72282ae73be01af8e7735: Status 404 returned error can't find the container with id fcb93d278b0e15550319f205617fe59147f6a7a955d72282ae73be01af8e7735
Sep 16 17:33:16.254898 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:16.254849    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:16.366556 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:16.366521    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:16.457377 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:16.457293    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:17.054640 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:17.054594    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-6fpjp" event={"ID":"5c1d21e6-c8e9-4321-89b0-a6246e89db0d","Type":"ContainerStarted","Data":"fcb93d278b0e15550319f205617fe59147f6a7a955d72282ae73be01af8e7735"}
Sep 16 17:33:18.473114 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:18.473068    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:18.605896 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:18.605832    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:20.071467 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:20.071422    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-v68f2" event={"ID":"1d7f4c40-b184-4988-9c18-27ab2c8ea02b","Type":"ContainerStarted","Data":"5a9122554890dc1d8022c83d481345c1e3e0bb8d9c3d79ca2130324179aecce9"}
Sep 16 17:33:20.071950 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:20.071497    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-v68f2"
Sep 16 17:33:20.072897 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:20.072851    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-6fpjp" event={"ID":"5c1d21e6-c8e9-4321-89b0-a6246e89db0d","Type":"ContainerStarted","Data":"f333e228ac9084f206aa893e815188a5aaea4dfeaf1c6d769ca90f22201040df"}
Sep 16 17:33:20.073028 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:20.073010    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-6fpjp"
Sep 16 17:33:20.100050 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:20.100009    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-v68f2" podStartSLOduration=2.293659787 podStartE2EDuration="6.099994466s" totalImagesPullingTime="3.806334679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:33:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:33:20.098784113 +0000 UTC m=+521.461462898" watchObservedRunningTime="2026-09-16 17:33:20.099994466 +0000 UTC m=+521.462673252"
Sep 16 17:33:20.124477 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:20.124424    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-6fpjp" podStartSLOduration=2.537192866 podStartE2EDuration="6.124404245s" totalImagesPullingTime="3.587211379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:33:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:33:20.123827383 +0000 UTC m=+521.486506169" watchObservedRunningTime="2026-09-16 17:33:20.124404245 +0000 UTC m=+521.487083035"
Sep 16 17:33:20.382831 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:20.382740    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:20.508176 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:20.508128    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:33:31.081692 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:31.081663    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-6fpjp"
Sep 16 17:33:36.049755 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:36.049720    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-p7vpt"
Sep 16 17:33:41.080032 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:33:41.079999    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-v68f2"
Sep 16 17:34:07.423337 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:07.423292    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:34:07.519419 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:07.519386    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:34:07.685534 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:07.685451    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tln2b"]
Sep 16 17:34:07.688347 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:07.688328    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tln2b"
Sep 16 17:34:07.691237 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:07.691213    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 17:34:07.691339 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:07.691262    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-8lpxq\""
Sep 16 17:34:07.692304 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:07.692286    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 17:34:07.699644 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:07.699623    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tln2b"]
Sep 16 17:34:07.759496 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:07.759454    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cabbd152-d691-4084-bf4d-5da61bc84131-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tln2b\" (UID: \"cabbd152-d691-4084-bf4d-5da61bc84131\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tln2b"
Sep 16 17:34:07.759496 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:07.759493    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cabbd152-d691-4084-bf4d-5da61bc84131-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tln2b\" (UID: \"cabbd152-d691-4084-bf4d-5da61bc84131\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tln2b"
Sep 16 17:34:07.759698 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:07.759575    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpldh\" (UniqueName: \"kubernetes.io/projected/cabbd152-d691-4084-bf4d-5da61bc84131-kube-api-access-kpldh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tln2b\" (UID: \"cabbd152-d691-4084-bf4d-5da61bc84131\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tln2b"
Sep 16 17:34:07.860871 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:07.860832    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpldh\" (UniqueName: \"kubernetes.io/projected/cabbd152-d691-4084-bf4d-5da61bc84131-kube-api-access-kpldh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tln2b\" (UID: \"cabbd152-d691-4084-bf4d-5da61bc84131\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tln2b"
Sep 16 17:34:07.860997 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:07.860925    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cabbd152-d691-4084-bf4d-5da61bc84131-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tln2b\" (UID: \"cabbd152-d691-4084-bf4d-5da61bc84131\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tln2b"
Sep 16 17:34:07.860997 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:07.860949    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cabbd152-d691-4084-bf4d-5da61bc84131-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tln2b\" (UID: \"cabbd152-d691-4084-bf4d-5da61bc84131\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tln2b"
Sep 16 17:34:07.861271 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:07.861258    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cabbd152-d691-4084-bf4d-5da61bc84131-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tln2b\" (UID: \"cabbd152-d691-4084-bf4d-5da61bc84131\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tln2b"
Sep 16 17:34:07.861342 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:07.861322    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cabbd152-d691-4084-bf4d-5da61bc84131-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tln2b\" (UID: \"cabbd152-d691-4084-bf4d-5da61bc84131\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tln2b"
Sep 16 17:34:07.870204 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:07.870181    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpldh\" (UniqueName: \"kubernetes.io/projected/cabbd152-d691-4084-bf4d-5da61bc84131-kube-api-access-kpldh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tln2b\" (UID: \"cabbd152-d691-4084-bf4d-5da61bc84131\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tln2b"
Sep 16 17:34:07.997824 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:07.997738    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tln2b"
Sep 16 17:34:08.276494 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:08.276470    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tln2b"]
Sep 16 17:34:08.280120 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:34:08.280086    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcabbd152_d691_4084_bf4d_5da61bc84131.slice/crio-4ca5e4af6125920c637b1f03f94d0c26b242ddad0a8959cb11a58a058a0d33ed WatchSource:0}: Error finding container 4ca5e4af6125920c637b1f03f94d0c26b242ddad0a8959cb11a58a058a0d33ed: Status 404 returned error can't find the container with id 4ca5e4af6125920c637b1f03f94d0c26b242ddad0a8959cb11a58a058a0d33ed
Sep 16 17:34:09.283183 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:09.283093    2577 generic.go:356] "Generic (PLEG): container finished" podID="cabbd152-d691-4084-bf4d-5da61bc84131" containerID="5ec47474db47fe0a621af4d60fc7d84dbd9fbc2e3aaa2e3565643ac2425a7e4a" exitCode=0
Sep 16 17:34:09.283183 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:09.283143    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tln2b" event={"ID":"cabbd152-d691-4084-bf4d-5da61bc84131","Type":"ContainerDied","Data":"5ec47474db47fe0a621af4d60fc7d84dbd9fbc2e3aaa2e3565643ac2425a7e4a"}
Sep 16 17:34:09.283685 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:09.283210    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tln2b" event={"ID":"cabbd152-d691-4084-bf4d-5da61bc84131","Type":"ContainerStarted","Data":"4ca5e4af6125920c637b1f03f94d0c26b242ddad0a8959cb11a58a058a0d33ed"}
Sep 16 17:34:12.307074 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:12.307034    2577 generic.go:356] "Generic (PLEG): container finished" podID="cabbd152-d691-4084-bf4d-5da61bc84131" containerID="bdfa2e1a93cb772faf488dcfefd7fbd9af1d023a7dafac66b59a7cb675b59c6e" exitCode=0
Sep 16 17:34:12.307489 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:12.307116    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tln2b" event={"ID":"cabbd152-d691-4084-bf4d-5da61bc84131","Type":"ContainerDied","Data":"bdfa2e1a93cb772faf488dcfefd7fbd9af1d023a7dafac66b59a7cb675b59c6e"}
Sep 16 17:34:13.313015 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:13.312976    2577 generic.go:356] "Generic (PLEG): container finished" podID="cabbd152-d691-4084-bf4d-5da61bc84131" containerID="1b01d182f39ba4aa261d79b9ea3069093094f07675b5646157e90486d097ee09" exitCode=0
Sep 16 17:34:13.313376 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:13.313036    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tln2b" event={"ID":"cabbd152-d691-4084-bf4d-5da61bc84131","Type":"ContainerDied","Data":"1b01d182f39ba4aa261d79b9ea3069093094f07675b5646157e90486d097ee09"}
Sep 16 17:34:14.443076 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:14.443051    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tln2b"
Sep 16 17:34:14.520884 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:14.520829    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cabbd152-d691-4084-bf4d-5da61bc84131-bundle\") pod \"cabbd152-d691-4084-bf4d-5da61bc84131\" (UID: \"cabbd152-d691-4084-bf4d-5da61bc84131\") "
Sep 16 17:34:14.521063 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:14.520896    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cabbd152-d691-4084-bf4d-5da61bc84131-util\") pod \"cabbd152-d691-4084-bf4d-5da61bc84131\" (UID: \"cabbd152-d691-4084-bf4d-5da61bc84131\") "
Sep 16 17:34:14.521063 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:14.520929    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpldh\" (UniqueName: \"kubernetes.io/projected/cabbd152-d691-4084-bf4d-5da61bc84131-kube-api-access-kpldh\") pod \"cabbd152-d691-4084-bf4d-5da61bc84131\" (UID: \"cabbd152-d691-4084-bf4d-5da61bc84131\") "
Sep 16 17:34:14.521604 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:14.521569    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cabbd152-d691-4084-bf4d-5da61bc84131-bundle" (OuterVolumeSpecName: "bundle") pod "cabbd152-d691-4084-bf4d-5da61bc84131" (UID: "cabbd152-d691-4084-bf4d-5da61bc84131"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:34:14.522979 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:14.522952    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cabbd152-d691-4084-bf4d-5da61bc84131-kube-api-access-kpldh" (OuterVolumeSpecName: "kube-api-access-kpldh") pod "cabbd152-d691-4084-bf4d-5da61bc84131" (UID: "cabbd152-d691-4084-bf4d-5da61bc84131"). InnerVolumeSpecName "kube-api-access-kpldh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:34:14.525514 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:14.525474    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cabbd152-d691-4084-bf4d-5da61bc84131-util" (OuterVolumeSpecName: "util") pod "cabbd152-d691-4084-bf4d-5da61bc84131" (UID: "cabbd152-d691-4084-bf4d-5da61bc84131"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:34:14.621776 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:14.621699    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cabbd152-d691-4084-bf4d-5da61bc84131-bundle\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:34:14.621776 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:14.621726    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cabbd152-d691-4084-bf4d-5da61bc84131-util\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:34:14.621776 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:14.621737    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpldh\" (UniqueName: \"kubernetes.io/projected/cabbd152-d691-4084-bf4d-5da61bc84131-kube-api-access-kpldh\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:34:15.323404 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:15.323371    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tln2b"
Sep 16 17:34:15.323561 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:15.323371    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tln2b" event={"ID":"cabbd152-d691-4084-bf4d-5da61bc84131","Type":"ContainerDied","Data":"4ca5e4af6125920c637b1f03f94d0c26b242ddad0a8959cb11a58a058a0d33ed"}
Sep 16 17:34:15.323561 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:15.323495    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ca5e4af6125920c637b1f03f94d0c26b242ddad0a8959cb11a58a058a0d33ed"
Sep 16 17:34:18.629943 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:18.629906    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:34:19.393451 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:19.393413    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:34:20.784782 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:20.784740    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-pgck7"]
Sep 16 17:34:20.785265 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:20.785152    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cabbd152-d691-4084-bf4d-5da61bc84131" containerName="util"
Sep 16 17:34:20.785265 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:20.785170    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cabbd152-d691-4084-bf4d-5da61bc84131" containerName="util"
Sep 16 17:34:20.785265 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:20.785195    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cabbd152-d691-4084-bf4d-5da61bc84131" containerName="pull"
Sep 16 17:34:20.785265 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:20.785203    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cabbd152-d691-4084-bf4d-5da61bc84131" containerName="pull"
Sep 16 17:34:20.785265 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:20.785212    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cabbd152-d691-4084-bf4d-5da61bc84131" containerName="extract"
Sep 16 17:34:20.785265 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:20.785222    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cabbd152-d691-4084-bf4d-5da61bc84131" containerName="extract"
Sep 16 17:34:20.785548 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:20.785294    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cabbd152-d691-4084-bf4d-5da61bc84131" containerName="extract"
Sep 16 17:34:20.789448 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:20.789427    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-pgck7"
Sep 16 17:34:20.793780 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:20.793759    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-p5j9z\""
Sep 16 17:34:20.794029 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:20.794013    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 16 17:34:20.794712 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:20.794695    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 16 17:34:20.806945 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:20.806920    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-pgck7"]
Sep 16 17:34:20.965165 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:20.965125    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h77q7\" (UniqueName: \"kubernetes.io/projected/7ca89a0c-fcc0-42fa-bb36-047873b06e6e-kube-api-access-h77q7\") pod \"cert-manager-operator-controller-manager-684f79c9b9-pgck7\" (UID: \"7ca89a0c-fcc0-42fa-bb36-047873b06e6e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-pgck7"
Sep 16 17:34:20.965165 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:20.965169    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ca89a0c-fcc0-42fa-bb36-047873b06e6e-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-pgck7\" (UID: \"7ca89a0c-fcc0-42fa-bb36-047873b06e6e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-pgck7"
Sep 16 17:34:21.066405 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:21.066375    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h77q7\" (UniqueName: \"kubernetes.io/projected/7ca89a0c-fcc0-42fa-bb36-047873b06e6e-kube-api-access-h77q7\") pod \"cert-manager-operator-controller-manager-684f79c9b9-pgck7\" (UID: \"7ca89a0c-fcc0-42fa-bb36-047873b06e6e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-pgck7"
Sep 16 17:34:21.066566 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:21.066417    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ca89a0c-fcc0-42fa-bb36-047873b06e6e-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-pgck7\" (UID: \"7ca89a0c-fcc0-42fa-bb36-047873b06e6e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-pgck7"
Sep 16 17:34:21.066798 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:21.066782    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ca89a0c-fcc0-42fa-bb36-047873b06e6e-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-pgck7\" (UID: \"7ca89a0c-fcc0-42fa-bb36-047873b06e6e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-pgck7"
Sep 16 17:34:21.075358 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:21.075327    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h77q7\" (UniqueName: \"kubernetes.io/projected/7ca89a0c-fcc0-42fa-bb36-047873b06e6e-kube-api-access-h77q7\") pod \"cert-manager-operator-controller-manager-684f79c9b9-pgck7\" (UID: \"7ca89a0c-fcc0-42fa-bb36-047873b06e6e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-pgck7"
Sep 16 17:34:21.098266 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:21.098242    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-pgck7"
Sep 16 17:34:21.381561 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:21.381531    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-pgck7"]
Sep 16 17:34:21.382630 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:34:21.382603    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca89a0c_fcc0_42fa_bb36_047873b06e6e.slice/crio-82bb02dd808fa6b551798c69dd535677cd4e5df494a210ed45ef3547f38141bc WatchSource:0}: Error finding container 82bb02dd808fa6b551798c69dd535677cd4e5df494a210ed45ef3547f38141bc: Status 404 returned error can't find the container with id 82bb02dd808fa6b551798c69dd535677cd4e5df494a210ed45ef3547f38141bc
Sep 16 17:34:22.357958 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:22.357921    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-pgck7" event={"ID":"7ca89a0c-fcc0-42fa-bb36-047873b06e6e","Type":"ContainerStarted","Data":"82bb02dd808fa6b551798c69dd535677cd4e5df494a210ed45ef3547f38141bc"}
Sep 16 17:34:22.418796 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:22.418764    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:34:22.517814 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:22.517770    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:34:26.378174 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:26.378136    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-pgck7" event={"ID":"7ca89a0c-fcc0-42fa-bb36-047873b06e6e","Type":"ContainerStarted","Data":"c487d563bb18369254df5c57423c36fd4fb923e1fd3270acda4f3c8cae8344bf"}
Sep 16 17:34:26.436312 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:26.436206    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-pgck7" podStartSLOduration=1.665906517 podStartE2EDuration="6.436192007s" totalImagesPullingTime="4.77028549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:34:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:34:26.435994138 +0000 UTC m=+587.798672924" watchObservedRunningTime="2026-09-16 17:34:26.436192007 +0000 UTC m=+587.798870796"
Sep 16 17:34:27.074230 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:27.074182    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:34:27.252162 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:27.252124    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:34:28.030920 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:28.030881    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:34:28.306550 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:28.306447    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:34:28.982139 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:28.982096    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:34:29.292615 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:29.292509    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:34:29.505620 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:29.505588    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:34:29.739119 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:29.739089    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:34:30.182820 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:30.182779    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:34:30.799647 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:30.799614    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwpnwc"]
Sep 16 17:34:30.803273 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:30.803251    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwpnwc"
Sep 16 17:34:30.806783 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:30.806764    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 17:34:30.806912 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:30.806859    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 17:34:30.807639 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:30.807623    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-8lpxq\""
Sep 16 17:34:30.807719 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:34:30.807617    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwpnwc\" is forbidden: User \"system:node:ip-10-0-136-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-marketplace\": no relationship found between node 'ip-10-0-136-122.ec2.internal' and this object" podUID="cc69c8cd-98d7-4c2e-aced-b3ea0d046c21" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwpnwc"
Sep 16 17:34:30.860193 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:30.860158    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwpnwc"]
Sep 16 17:34:30.946610 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:30.946575    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24thf\" (UniqueName: \"kubernetes.io/projected/cc69c8cd-98d7-4c2e-aced-b3ea0d046c21-kube-api-access-24thf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwpnwc\" (UID: \"cc69c8cd-98d7-4c2e-aced-b3ea0d046c21\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwpnwc"
Sep 16 17:34:30.946610 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:30.946616    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc69c8cd-98d7-4c2e-aced-b3ea0d046c21-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwpnwc\" (UID: \"cc69c8cd-98d7-4c2e-aced-b3ea0d046c21\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwpnwc"
Sep 16 17:34:30.946888 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:30.946639    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc69c8cd-98d7-4c2e-aced-b3ea0d046c21-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwpnwc\" (UID: \"cc69c8cd-98d7-4c2e-aced-b3ea0d046c21\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwpnwc"
Sep 16 17:34:31.047559 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:31.047522    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24thf\" (UniqueName: \"kubernetes.io/projected/cc69c8cd-98d7-4c2e-aced-b3ea0d046c21-kube-api-access-24thf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwpnwc\" (UID: \"cc69c8cd-98d7-4c2e-aced-b3ea0d046c21\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwpnwc"
Sep 16 17:34:31.047559 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:31.047565    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc69c8cd-98d7-4c2e-aced-b3ea0d046c21-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwpnwc\" (UID: \"cc69c8cd-98d7-4c2e-aced-b3ea0d046c21\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwpnwc"
Sep 16 17:34:31.047775 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:31.047585    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc69c8cd-98d7-4c2e-aced-b3ea0d046c21-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwpnwc\" (UID: \"cc69c8cd-98d7-4c2e-aced-b3ea0d046c21\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwpnwc"
Sep 16 17:34:31.048017 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:31.047996    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc69c8cd-98d7-4c2e-aced-b3ea0d046c21-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwpnwc\" (UID: \"cc69c8cd-98d7-4c2e-aced-b3ea0d046c21\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwpnwc"
Sep 16 17:34:31.048059 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:31.048016    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc69c8cd-98d7-4c2e-aced-b3ea0d046c21-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwpnwc\" (UID: \"cc69c8cd-98d7-4c2e-aced-b3ea0d046c21\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwpnwc"
Sep 16 17:34:31.082042 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:31.082017    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24thf\" (UniqueName: \"kubernetes.io/projected/cc69c8cd-98d7-4c2e-aced-b3ea0d046c21-kube-api-access-24thf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwpnwc\" (UID: \"cc69c8cd-98d7-4c2e-aced-b3ea0d046c21\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwpnwc"
Sep 16 17:34:31.111503 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:31.111479    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwpnwc"
Sep 16 17:34:31.407720 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:31.407633    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwpnwc"]
Sep 16 17:34:31.411805 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:34:31.411777    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc69c8cd_98d7_4c2e_aced_b3ea0d046c21.slice/crio-673181e67dbc6600eaa9b19f0951aee8fcf58639a77616b4993cdc995e4581e7 WatchSource:0}: Error finding container 673181e67dbc6600eaa9b19f0951aee8fcf58639a77616b4993cdc995e4581e7: Status 404 returned error can't find the container with id 673181e67dbc6600eaa9b19f0951aee8fcf58639a77616b4993cdc995e4581e7
Sep 16 17:34:32.403538 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:32.403448    2577 generic.go:356] "Generic (PLEG): container finished" podID="cc69c8cd-98d7-4c2e-aced-b3ea0d046c21" containerID="19bc7833b001becf9d50ec90261dc5d396182f9f51be6f17f5f8640d8e9bdd5c" exitCode=0
Sep 16 17:34:32.403905 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:32.403536    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwpnwc" event={"ID":"cc69c8cd-98d7-4c2e-aced-b3ea0d046c21","Type":"ContainerDied","Data":"19bc7833b001becf9d50ec90261dc5d396182f9f51be6f17f5f8640d8e9bdd5c"}
Sep 16 17:34:32.403905 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:32.403592    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwpnwc" event={"ID":"cc69c8cd-98d7-4c2e-aced-b3ea0d046c21","Type":"ContainerStarted","Data":"673181e67dbc6600eaa9b19f0951aee8fcf58639a77616b4993cdc995e4581e7"}
Sep 16 17:34:34.414330 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:34.414298    2577 generic.go:356] "Generic (PLEG): container finished" podID="cc69c8cd-98d7-4c2e-aced-b3ea0d046c21" containerID="3b496b6e80c55d88603fdea2e69a30860b78c7ef9496371d60641898fc0865fe" exitCode=0
Sep 16 17:34:34.414682 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:34.414386    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwpnwc" event={"ID":"cc69c8cd-98d7-4c2e-aced-b3ea0d046c21","Type":"ContainerDied","Data":"3b496b6e80c55d88603fdea2e69a30860b78c7ef9496371d60641898fc0865fe"}
Sep 16 17:34:35.420712 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:35.420682    2577 generic.go:356] "Generic (PLEG): container finished" podID="cc69c8cd-98d7-4c2e-aced-b3ea0d046c21" containerID="03f420fff12cbbefc6491f97c2185a2fc1154c6bd3cc385cd8eac640ac860f0b" exitCode=0
Sep 16 17:34:35.421104 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:35.420772    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwpnwc" event={"ID":"cc69c8cd-98d7-4c2e-aced-b3ea0d046c21","Type":"ContainerDied","Data":"03f420fff12cbbefc6491f97c2185a2fc1154c6bd3cc385cd8eac640ac860f0b"}
Sep 16 17:34:36.543839 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:36.543795    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwpnwc"
Sep 16 17:34:36.595295 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:36.595257    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24thf\" (UniqueName: \"kubernetes.io/projected/cc69c8cd-98d7-4c2e-aced-b3ea0d046c21-kube-api-access-24thf\") pod \"cc69c8cd-98d7-4c2e-aced-b3ea0d046c21\" (UID: \"cc69c8cd-98d7-4c2e-aced-b3ea0d046c21\") "
Sep 16 17:34:36.595295 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:36.595296    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc69c8cd-98d7-4c2e-aced-b3ea0d046c21-util\") pod \"cc69c8cd-98d7-4c2e-aced-b3ea0d046c21\" (UID: \"cc69c8cd-98d7-4c2e-aced-b3ea0d046c21\") "
Sep 16 17:34:36.595516 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:36.595389    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc69c8cd-98d7-4c2e-aced-b3ea0d046c21-bundle\") pod \"cc69c8cd-98d7-4c2e-aced-b3ea0d046c21\" (UID: \"cc69c8cd-98d7-4c2e-aced-b3ea0d046c21\") "
Sep 16 17:34:36.595771 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:36.595744    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc69c8cd-98d7-4c2e-aced-b3ea0d046c21-bundle" (OuterVolumeSpecName: "bundle") pod "cc69c8cd-98d7-4c2e-aced-b3ea0d046c21" (UID: "cc69c8cd-98d7-4c2e-aced-b3ea0d046c21"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:34:36.597416 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:36.597386    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc69c8cd-98d7-4c2e-aced-b3ea0d046c21-kube-api-access-24thf" (OuterVolumeSpecName: "kube-api-access-24thf") pod "cc69c8cd-98d7-4c2e-aced-b3ea0d046c21" (UID: "cc69c8cd-98d7-4c2e-aced-b3ea0d046c21"). InnerVolumeSpecName "kube-api-access-24thf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:34:36.600766 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:36.600745    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc69c8cd-98d7-4c2e-aced-b3ea0d046c21-util" (OuterVolumeSpecName: "util") pod "cc69c8cd-98d7-4c2e-aced-b3ea0d046c21" (UID: "cc69c8cd-98d7-4c2e-aced-b3ea0d046c21"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:34:36.696447 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:36.696357    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc69c8cd-98d7-4c2e-aced-b3ea0d046c21-bundle\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:34:36.696447 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:36.696387    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24thf\" (UniqueName: \"kubernetes.io/projected/cc69c8cd-98d7-4c2e-aced-b3ea0d046c21-kube-api-access-24thf\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:34:36.696447 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:36.696402    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc69c8cd-98d7-4c2e-aced-b3ea0d046c21-util\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:34:36.867583 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:36.867554    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:34:36.968546 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:36.968462    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:34:37.431083 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:37.431049    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwpnwc" event={"ID":"cc69c8cd-98d7-4c2e-aced-b3ea0d046c21","Type":"ContainerDied","Data":"673181e67dbc6600eaa9b19f0951aee8fcf58639a77616b4993cdc995e4581e7"}
Sep 16 17:34:37.431226 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:37.431094    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="673181e67dbc6600eaa9b19f0951aee8fcf58639a77616b4993cdc995e4581e7"
Sep 16 17:34:37.431226 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:37.431101    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwpnwc"
Sep 16 17:34:37.596738 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:37.596679    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:34:41.616211 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:41.616175    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:34:41.715597 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:41.715558    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:34:42.944124 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:42.944077    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zzmqd"]
Sep 16 17:34:42.944716 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:42.944493    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc69c8cd-98d7-4c2e-aced-b3ea0d046c21" containerName="util"
Sep 16 17:34:42.944716 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:42.944510    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc69c8cd-98d7-4c2e-aced-b3ea0d046c21" containerName="util"
Sep 16 17:34:42.944716 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:42.944524    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc69c8cd-98d7-4c2e-aced-b3ea0d046c21" containerName="extract"
Sep 16 17:34:42.944716 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:42.944532    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc69c8cd-98d7-4c2e-aced-b3ea0d046c21" containerName="extract"
Sep 16 17:34:42.944716 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:42.944562    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc69c8cd-98d7-4c2e-aced-b3ea0d046c21" containerName="pull"
Sep 16 17:34:42.944716 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:42.944570    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc69c8cd-98d7-4c2e-aced-b3ea0d046c21" containerName="pull"
Sep 16 17:34:42.944716 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:42.944670    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc69c8cd-98d7-4c2e-aced-b3ea0d046c21" containerName="extract"
Sep 16 17:34:42.947477 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:42.947455    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zzmqd"
Sep 16 17:34:42.950002 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:42.949985    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 17:34:42.950799 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:42.950784    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 17:34:42.950897 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:42.950840    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-4fxgv\""
Sep 16 17:34:42.957360 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:42.957337    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zzmqd"]
Sep 16 17:34:43.043182 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:43.043147    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wdm4\" (UniqueName: \"kubernetes.io/projected/04df58f0-1d3e-486f-ae6b-5ab7f27b4596-kube-api-access-7wdm4\") pod \"openshift-lws-operator-fd8ccff4c-zzmqd\" (UID: \"04df58f0-1d3e-486f-ae6b-5ab7f27b4596\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zzmqd"
Sep 16 17:34:43.043182 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:43.043192    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04df58f0-1d3e-486f-ae6b-5ab7f27b4596-tmp\") pod \"openshift-lws-operator-fd8ccff4c-zzmqd\" (UID: \"04df58f0-1d3e-486f-ae6b-5ab7f27b4596\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zzmqd"
Sep 16 17:34:43.144330 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:43.144290    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wdm4\" (UniqueName: \"kubernetes.io/projected/04df58f0-1d3e-486f-ae6b-5ab7f27b4596-kube-api-access-7wdm4\") pod \"openshift-lws-operator-fd8ccff4c-zzmqd\" (UID: \"04df58f0-1d3e-486f-ae6b-5ab7f27b4596\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zzmqd"
Sep 16 17:34:43.144526 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:43.144341    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04df58f0-1d3e-486f-ae6b-5ab7f27b4596-tmp\") pod \"openshift-lws-operator-fd8ccff4c-zzmqd\" (UID: \"04df58f0-1d3e-486f-ae6b-5ab7f27b4596\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zzmqd"
Sep 16 17:34:43.144756 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:43.144731    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04df58f0-1d3e-486f-ae6b-5ab7f27b4596-tmp\") pod \"openshift-lws-operator-fd8ccff4c-zzmqd\" (UID: \"04df58f0-1d3e-486f-ae6b-5ab7f27b4596\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zzmqd"
Sep 16 17:34:43.153592 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:43.153564    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wdm4\" (UniqueName: \"kubernetes.io/projected/04df58f0-1d3e-486f-ae6b-5ab7f27b4596-kube-api-access-7wdm4\") pod \"openshift-lws-operator-fd8ccff4c-zzmqd\" (UID: \"04df58f0-1d3e-486f-ae6b-5ab7f27b4596\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zzmqd"
Sep 16 17:34:43.256732 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:43.256635    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zzmqd"
Sep 16 17:34:43.528654 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:43.528629    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zzmqd"]
Sep 16 17:34:43.530947 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:34:43.530919    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04df58f0_1d3e_486f_ae6b_5ab7f27b4596.slice/crio-5ae51fe2628497999a775c4ef6c1192ac6daf0f8379496909348351904f0c2b4 WatchSource:0}: Error finding container 5ae51fe2628497999a775c4ef6c1192ac6daf0f8379496909348351904f0c2b4: Status 404 returned error can't find the container with id 5ae51fe2628497999a775c4ef6c1192ac6daf0f8379496909348351904f0c2b4
Sep 16 17:34:44.464891 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:44.464838    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zzmqd" event={"ID":"04df58f0-1d3e-486f-ae6b-5ab7f27b4596","Type":"ContainerStarted","Data":"5ae51fe2628497999a775c4ef6c1192ac6daf0f8379496909348351904f0c2b4"}
Sep 16 17:34:45.470828 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:45.470738    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zzmqd" event={"ID":"04df58f0-1d3e-486f-ae6b-5ab7f27b4596","Type":"ContainerStarted","Data":"b3344796efb06e9577fc6f155b02d57d3556d3fe0777012d282812fc59e93407"}
Sep 16 17:34:45.521986 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:45.521933    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-zzmqd" podStartSLOduration=1.991944169 podStartE2EDuration="3.521916424s" totalImagesPullingTime="1.529972255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:34:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:34:45.521449721 +0000 UTC m=+606.884128508" watchObservedRunningTime="2026-09-16 17:34:45.521916424 +0000 UTC m=+606.884595209"
Sep 16 17:34:52.783977 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:52.783934    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:34:57.704630 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:57.704590    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:34:57.705055 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:57.704943    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c"]
Sep 16 17:34:57.708091 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:57.708075    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c"
Sep 16 17:34:57.711244 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:57.711223    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 16 17:34:57.711836 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:57.711818    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 16 17:34:57.711953 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:57.711921    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-hg6z6\""
Sep 16 17:34:57.712262 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:57.712248    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 17:34:57.712332 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:57.712250    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 16 17:34:57.724986 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:57.724961    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c"]
Sep 16 17:34:57.771563 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:57.771522    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d3482ef2-f205-428e-b2fa-0019c70fed06-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cp95c\" (UID: \"d3482ef2-f205-428e-b2fa-0019c70fed06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c"
Sep 16 17:34:57.771563 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:57.771567    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3482ef2-f205-428e-b2fa-0019c70fed06-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cp95c\" (UID: \"d3482ef2-f205-428e-b2fa-0019c70fed06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c"
Sep 16 17:34:57.771785 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:57.771654    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3482ef2-f205-428e-b2fa-0019c70fed06-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cp95c\" (UID: \"d3482ef2-f205-428e-b2fa-0019c70fed06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c"
Sep 16 17:34:57.771785 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:57.771678    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d3482ef2-f205-428e-b2fa-0019c70fed06-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cp95c\" (UID: \"d3482ef2-f205-428e-b2fa-0019c70fed06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c"
Sep 16 17:34:57.771785 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:57.771701    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd265\" (UniqueName: \"kubernetes.io/projected/d3482ef2-f205-428e-b2fa-0019c70fed06-kube-api-access-xd265\") pod \"istiod-openshift-gateway-7b7995d5cd-cp95c\" (UID: \"d3482ef2-f205-428e-b2fa-0019c70fed06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c"
Sep 16 17:34:57.771785 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:57.771725    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d3482ef2-f205-428e-b2fa-0019c70fed06-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cp95c\" (UID: \"d3482ef2-f205-428e-b2fa-0019c70fed06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c"
Sep 16 17:34:57.771785 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:57.771758    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d3482ef2-f205-428e-b2fa-0019c70fed06-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cp95c\" (UID: \"d3482ef2-f205-428e-b2fa-0019c70fed06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c"
Sep 16 17:34:57.805606 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:57.805561    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:34:57.872758 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:57.872723    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3482ef2-f205-428e-b2fa-0019c70fed06-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cp95c\" (UID: \"d3482ef2-f205-428e-b2fa-0019c70fed06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c"
Sep 16 17:34:57.872758 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:57.872756    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d3482ef2-f205-428e-b2fa-0019c70fed06-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cp95c\" (UID: \"d3482ef2-f205-428e-b2fa-0019c70fed06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c"
Sep 16 17:34:57.873031 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:57.872777    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd265\" (UniqueName: \"kubernetes.io/projected/d3482ef2-f205-428e-b2fa-0019c70fed06-kube-api-access-xd265\") pod \"istiod-openshift-gateway-7b7995d5cd-cp95c\" (UID: \"d3482ef2-f205-428e-b2fa-0019c70fed06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c"
Sep 16 17:34:57.873031 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:57.872810    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d3482ef2-f205-428e-b2fa-0019c70fed06-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cp95c\" (UID: \"d3482ef2-f205-428e-b2fa-0019c70fed06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c"
Sep 16 17:34:57.873031 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:57.872834    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d3482ef2-f205-428e-b2fa-0019c70fed06-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cp95c\" (UID: \"d3482ef2-f205-428e-b2fa-0019c70fed06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c"
Sep 16 17:34:57.873031 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:57.872907    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d3482ef2-f205-428e-b2fa-0019c70fed06-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cp95c\" (UID: \"d3482ef2-f205-428e-b2fa-0019c70fed06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c"
Sep 16 17:34:57.873031 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:57.872940    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3482ef2-f205-428e-b2fa-0019c70fed06-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cp95c\" (UID: \"d3482ef2-f205-428e-b2fa-0019c70fed06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c"
Sep 16 17:34:57.873594 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:57.873554    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d3482ef2-f205-428e-b2fa-0019c70fed06-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cp95c\" (UID: \"d3482ef2-f205-428e-b2fa-0019c70fed06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c"
Sep 16 17:34:57.875316 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:57.875289    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d3482ef2-f205-428e-b2fa-0019c70fed06-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cp95c\" (UID: \"d3482ef2-f205-428e-b2fa-0019c70fed06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c"
Sep 16 17:34:57.875430 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:57.875406    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d3482ef2-f205-428e-b2fa-0019c70fed06-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cp95c\" (UID: \"d3482ef2-f205-428e-b2fa-0019c70fed06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c"
Sep 16 17:34:57.875590 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:57.875571    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3482ef2-f205-428e-b2fa-0019c70fed06-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cp95c\" (UID: \"d3482ef2-f205-428e-b2fa-0019c70fed06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c"
Sep 16 17:34:57.875652 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:57.875604    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d3482ef2-f205-428e-b2fa-0019c70fed06-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cp95c\" (UID: \"d3482ef2-f205-428e-b2fa-0019c70fed06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c"
Sep 16 17:34:57.883429 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:57.883409    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd265\" (UniqueName: \"kubernetes.io/projected/d3482ef2-f205-428e-b2fa-0019c70fed06-kube-api-access-xd265\") pod \"istiod-openshift-gateway-7b7995d5cd-cp95c\" (UID: \"d3482ef2-f205-428e-b2fa-0019c70fed06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c"
Sep 16 17:34:57.884369 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:57.884350    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3482ef2-f205-428e-b2fa-0019c70fed06-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cp95c\" (UID: \"d3482ef2-f205-428e-b2fa-0019c70fed06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c"
Sep 16 17:34:58.017221 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:58.017138    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c"
Sep 16 17:34:58.296576 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:58.296463    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c"]
Sep 16 17:34:58.299317 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:34:58.299288    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3482ef2_f205_428e_b2fa_0019c70fed06.slice/crio-5bf47333bbf62537fd4704cfc3e7bdb10ecb62f1cb38f775a45935de935a47b5 WatchSource:0}: Error finding container 5bf47333bbf62537fd4704cfc3e7bdb10ecb62f1cb38f775a45935de935a47b5: Status 404 returned error can't find the container with id 5bf47333bbf62537fd4704cfc3e7bdb10ecb62f1cb38f775a45935de935a47b5
Sep 16 17:34:58.530626 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:58.530585    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c" event={"ID":"d3482ef2-f205-428e-b2fa-0019c70fed06","Type":"ContainerStarted","Data":"5bf47333bbf62537fd4704cfc3e7bdb10ecb62f1cb38f775a45935de935a47b5"}
Sep 16 17:34:58.938255 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:34:58.938221    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:00.759572 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:00.759520    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 17:35:00.759805 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:00.759609    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 17:35:01.547081 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:01.546999    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c" event={"ID":"d3482ef2-f205-428e-b2fa-0019c70fed06","Type":"ContainerStarted","Data":"b372296129f74cf2fbd862cae8087eb626601068f4f9c95bd26f0ab7b078b92b"}
Sep 16 17:35:01.547245 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:01.547213    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c"
Sep 16 17:35:01.549198 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:01.549171    2577 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-cp95c container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 17:35:01.549313 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:01.549217    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c" podUID="d3482ef2-f205-428e-b2fa-0019c70fed06" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:35:01.587003 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:01.586950    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c" podStartSLOduration=2.128810157 podStartE2EDuration="4.586936025s" totalImagesPullingTime="2.458125868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:34:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:35:01.583650901 +0000 UTC m=+622.946329689" watchObservedRunningTime="2026-09-16 17:35:01.586936025 +0000 UTC m=+622.949614812"
Sep 16 17:35:01.866719 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:01.866680    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:02.035732 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:02.035699    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:02.552305 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:02.552277    2577 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-cp95c container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 17:35:02.552462 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:02.552325    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c" podUID="d3482ef2-f205-428e-b2fa-0019c70fed06" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:35:05.552771 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:05.552736    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c"
Sep 16 17:35:06.220907 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:06.220828    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:06.325296 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:06.325251    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:10.200150 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:10.200112    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:11.721799 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:11.721762    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:11.830931 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:11.830899    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:12.437476 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:12.437443    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:12.648460 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:12.648417    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:17.159592 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.159549    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8vpf"]
Sep 16 17:35:17.168239 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.168213    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8vpf"
Sep 16 17:35:17.171120 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.171094    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 17:35:17.171250 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.171236    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-8lpxq\""
Sep 16 17:35:17.172458 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.172434    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 17:35:17.178406 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.178385    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8vpf"]
Sep 16 17:35:17.241848 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.241814    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vjgf\" (UniqueName: \"kubernetes.io/projected/50d32d47-f631-4886-84bd-309cb9dfaccb-kube-api-access-5vjgf\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8vpf\" (UID: \"50d32d47-f631-4886-84bd-309cb9dfaccb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8vpf"
Sep 16 17:35:17.242017 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.241890    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50d32d47-f631-4886-84bd-309cb9dfaccb-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8vpf\" (UID: \"50d32d47-f631-4886-84bd-309cb9dfaccb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8vpf"
Sep 16 17:35:17.242017 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.241926    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50d32d47-f631-4886-84bd-309cb9dfaccb-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8vpf\" (UID: \"50d32d47-f631-4886-84bd-309cb9dfaccb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8vpf"
Sep 16 17:35:17.271360 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.271326    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcxq4"]
Sep 16 17:35:17.275347 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.275325    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcxq4"
Sep 16 17:35:17.285479 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.285456    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcxq4"]
Sep 16 17:35:17.343120 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.343085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50d32d47-f631-4886-84bd-309cb9dfaccb-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8vpf\" (UID: \"50d32d47-f631-4886-84bd-309cb9dfaccb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8vpf"
Sep 16 17:35:17.343120 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.343124    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50d32d47-f631-4886-84bd-309cb9dfaccb-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8vpf\" (UID: \"50d32d47-f631-4886-84bd-309cb9dfaccb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8vpf"
Sep 16 17:35:17.343313 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.343149    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86fd1af6-b313-41cf-ba86-7676169148f5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcxq4\" (UID: \"86fd1af6-b313-41cf-ba86-7676169148f5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcxq4"
Sep 16 17:35:17.343313 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.343177    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s9cr\" (UniqueName: \"kubernetes.io/projected/86fd1af6-b313-41cf-ba86-7676169148f5-kube-api-access-9s9cr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcxq4\" (UID: \"86fd1af6-b313-41cf-ba86-7676169148f5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcxq4"
Sep 16 17:35:17.343313 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.343229    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vjgf\" (UniqueName: \"kubernetes.io/projected/50d32d47-f631-4886-84bd-309cb9dfaccb-kube-api-access-5vjgf\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8vpf\" (UID: \"50d32d47-f631-4886-84bd-309cb9dfaccb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8vpf"
Sep 16 17:35:17.343313 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.343263    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86fd1af6-b313-41cf-ba86-7676169148f5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcxq4\" (UID: \"86fd1af6-b313-41cf-ba86-7676169148f5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcxq4"
Sep 16 17:35:17.343463 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.343440    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50d32d47-f631-4886-84bd-309cb9dfaccb-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8vpf\" (UID: \"50d32d47-f631-4886-84bd-309cb9dfaccb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8vpf"
Sep 16 17:35:17.343502 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.343474    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50d32d47-f631-4886-84bd-309cb9dfaccb-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8vpf\" (UID: \"50d32d47-f631-4886-84bd-309cb9dfaccb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8vpf"
Sep 16 17:35:17.352310 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.352285    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9ssg"]
Sep 16 17:35:17.356265 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.356249    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9ssg"
Sep 16 17:35:17.356844 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.356824    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vjgf\" (UniqueName: \"kubernetes.io/projected/50d32d47-f631-4886-84bd-309cb9dfaccb-kube-api-access-5vjgf\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8vpf\" (UID: \"50d32d47-f631-4886-84bd-309cb9dfaccb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8vpf"
Sep 16 17:35:17.371687 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.371023    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9ssg"]
Sep 16 17:35:17.443794 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.443702    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86fd1af6-b313-41cf-ba86-7676169148f5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcxq4\" (UID: \"86fd1af6-b313-41cf-ba86-7676169148f5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcxq4"
Sep 16 17:35:17.443794 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.443744    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s9cr\" (UniqueName: \"kubernetes.io/projected/86fd1af6-b313-41cf-ba86-7676169148f5-kube-api-access-9s9cr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcxq4\" (UID: \"86fd1af6-b313-41cf-ba86-7676169148f5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcxq4"
Sep 16 17:35:17.443794 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.443765    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aeb7f372-1946-49ec-8599-d4db22b63d96-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9ssg\" (UID: \"aeb7f372-1946-49ec-8599-d4db22b63d96\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9ssg"
Sep 16 17:35:17.444124 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.443917    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjvng\" (UniqueName: \"kubernetes.io/projected/aeb7f372-1946-49ec-8599-d4db22b63d96-kube-api-access-gjvng\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9ssg\" (UID: \"aeb7f372-1946-49ec-8599-d4db22b63d96\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9ssg"
Sep 16 17:35:17.444124 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.443994    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86fd1af6-b313-41cf-ba86-7676169148f5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcxq4\" (UID: \"86fd1af6-b313-41cf-ba86-7676169148f5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcxq4"
Sep 16 17:35:17.444124 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.444093    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aeb7f372-1946-49ec-8599-d4db22b63d96-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9ssg\" (UID: \"aeb7f372-1946-49ec-8599-d4db22b63d96\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9ssg"
Sep 16 17:35:17.444248 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.444168    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86fd1af6-b313-41cf-ba86-7676169148f5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcxq4\" (UID: \"86fd1af6-b313-41cf-ba86-7676169148f5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcxq4"
Sep 16 17:35:17.444324 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.444304    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86fd1af6-b313-41cf-ba86-7676169148f5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcxq4\" (UID: \"86fd1af6-b313-41cf-ba86-7676169148f5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcxq4"
Sep 16 17:35:17.463616 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.463592    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qp4gp"]
Sep 16 17:35:17.463817 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.463799    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s9cr\" (UniqueName: \"kubernetes.io/projected/86fd1af6-b313-41cf-ba86-7676169148f5-kube-api-access-9s9cr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcxq4\" (UID: \"86fd1af6-b313-41cf-ba86-7676169148f5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcxq4"
Sep 16 17:35:17.467592 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.467577    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qp4gp"
Sep 16 17:35:17.478376 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.478352    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8vpf"
Sep 16 17:35:17.480388 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.480370    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qp4gp"]
Sep 16 17:35:17.544583 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.544550    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjvng\" (UniqueName: \"kubernetes.io/projected/aeb7f372-1946-49ec-8599-d4db22b63d96-kube-api-access-gjvng\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9ssg\" (UID: \"aeb7f372-1946-49ec-8599-d4db22b63d96\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9ssg"
Sep 16 17:35:17.544750 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.544595    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shqbx\" (UniqueName: \"kubernetes.io/projected/47a51c35-472e-41a7-a21b-67f12670cc8c-kube-api-access-shqbx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qp4gp\" (UID: \"47a51c35-472e-41a7-a21b-67f12670cc8c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qp4gp"
Sep 16 17:35:17.544750 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.544667    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47a51c35-472e-41a7-a21b-67f12670cc8c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qp4gp\" (UID: \"47a51c35-472e-41a7-a21b-67f12670cc8c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qp4gp"
Sep 16 17:35:17.544750 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.544693    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47a51c35-472e-41a7-a21b-67f12670cc8c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qp4gp\" (UID: \"47a51c35-472e-41a7-a21b-67f12670cc8c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qp4gp"
Sep 16 17:35:17.544750 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.544727    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aeb7f372-1946-49ec-8599-d4db22b63d96-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9ssg\" (UID: \"aeb7f372-1946-49ec-8599-d4db22b63d96\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9ssg"
Sep 16 17:35:17.544980 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.544777    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aeb7f372-1946-49ec-8599-d4db22b63d96-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9ssg\" (UID: \"aeb7f372-1946-49ec-8599-d4db22b63d96\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9ssg"
Sep 16 17:35:17.545228 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.545202    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aeb7f372-1946-49ec-8599-d4db22b63d96-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9ssg\" (UID: \"aeb7f372-1946-49ec-8599-d4db22b63d96\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9ssg"
Sep 16 17:35:17.545398 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.545377    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aeb7f372-1946-49ec-8599-d4db22b63d96-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9ssg\" (UID: \"aeb7f372-1946-49ec-8599-d4db22b63d96\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9ssg"
Sep 16 17:35:17.554743 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.554695    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjvng\" (UniqueName: \"kubernetes.io/projected/aeb7f372-1946-49ec-8599-d4db22b63d96-kube-api-access-gjvng\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9ssg\" (UID: \"aeb7f372-1946-49ec-8599-d4db22b63d96\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9ssg"
Sep 16 17:35:17.584650 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.584623    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcxq4"
Sep 16 17:35:17.645685 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.645651    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shqbx\" (UniqueName: \"kubernetes.io/projected/47a51c35-472e-41a7-a21b-67f12670cc8c-kube-api-access-shqbx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qp4gp\" (UID: \"47a51c35-472e-41a7-a21b-67f12670cc8c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qp4gp"
Sep 16 17:35:17.645822 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.645776    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47a51c35-472e-41a7-a21b-67f12670cc8c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qp4gp\" (UID: \"47a51c35-472e-41a7-a21b-67f12670cc8c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qp4gp"
Sep 16 17:35:17.645822 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.645814    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47a51c35-472e-41a7-a21b-67f12670cc8c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qp4gp\" (UID: \"47a51c35-472e-41a7-a21b-67f12670cc8c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qp4gp"
Sep 16 17:35:17.646329 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.646303    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47a51c35-472e-41a7-a21b-67f12670cc8c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qp4gp\" (UID: \"47a51c35-472e-41a7-a21b-67f12670cc8c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qp4gp"
Sep 16 17:35:17.646426 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.646347    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47a51c35-472e-41a7-a21b-67f12670cc8c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qp4gp\" (UID: \"47a51c35-472e-41a7-a21b-67f12670cc8c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qp4gp"
Sep 16 17:35:17.654657 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.654629    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shqbx\" (UniqueName: \"kubernetes.io/projected/47a51c35-472e-41a7-a21b-67f12670cc8c-kube-api-access-shqbx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qp4gp\" (UID: \"47a51c35-472e-41a7-a21b-67f12670cc8c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qp4gp"
Sep 16 17:35:17.678191 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.678165    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9ssg"
Sep 16 17:35:17.755047 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.755021    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8vpf"]
Sep 16 17:35:17.756758 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:35:17.756729    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50d32d47_f631_4886_84bd_309cb9dfaccb.slice/crio-c5e99b11b5ea1d9eb0c2d355cf889ac2702822cde24967df6d51c9329a396360 WatchSource:0}: Error finding container c5e99b11b5ea1d9eb0c2d355cf889ac2702822cde24967df6d51c9329a396360: Status 404 returned error can't find the container with id c5e99b11b5ea1d9eb0c2d355cf889ac2702822cde24967df6d51c9329a396360
Sep 16 17:35:17.777140 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.777119    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qp4gp"
Sep 16 17:35:17.859743 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.859714    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcxq4"]
Sep 16 17:35:17.900757 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:35:17.900725    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86fd1af6_b313_41cf_ba86_7676169148f5.slice/crio-c671c89cc603f423bcadfe87ced9b02577c8a096711674ba649d3451ffe67b27 WatchSource:0}: Error finding container c671c89cc603f423bcadfe87ced9b02577c8a096711674ba649d3451ffe67b27: Status 404 returned error can't find the container with id c671c89cc603f423bcadfe87ced9b02577c8a096711674ba649d3451ffe67b27
Sep 16 17:35:17.952348 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:17.952327    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9ssg"]
Sep 16 17:35:17.954738 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:35:17.954713    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeb7f372_1946_49ec_8599_d4db22b63d96.slice/crio-c4ef219b836f8c8b6755cf3c571cdd4ae85ee1902a904988ab424b82c0c5bb1d WatchSource:0}: Error finding container c4ef219b836f8c8b6755cf3c571cdd4ae85ee1902a904988ab424b82c0c5bb1d: Status 404 returned error can't find the container with id c4ef219b836f8c8b6755cf3c571cdd4ae85ee1902a904988ab424b82c0c5bb1d
Sep 16 17:35:18.050369 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:18.050330    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qp4gp"]
Sep 16 17:35:18.107810 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:35:18.107779    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47a51c35_472e_41a7_a21b_67f12670cc8c.slice/crio-6bc039e7d056fe6e52b73730105b3329196dcff4e813a876e1539c9b18ef7657 WatchSource:0}: Error finding container 6bc039e7d056fe6e52b73730105b3329196dcff4e813a876e1539c9b18ef7657: Status 404 returned error can't find the container with id 6bc039e7d056fe6e52b73730105b3329196dcff4e813a876e1539c9b18ef7657
Sep 16 17:35:18.619273 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:18.619233    2577 generic.go:356] "Generic (PLEG): container finished" podID="86fd1af6-b313-41cf-ba86-7676169148f5" containerID="fe0644848309919f28d239661489428afe6c734c7f4b48f22fa40e66351194f7" exitCode=0
Sep 16 17:35:18.619712 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:18.619318    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcxq4" event={"ID":"86fd1af6-b313-41cf-ba86-7676169148f5","Type":"ContainerDied","Data":"fe0644848309919f28d239661489428afe6c734c7f4b48f22fa40e66351194f7"}
Sep 16 17:35:18.619712 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:18.619371    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcxq4" event={"ID":"86fd1af6-b313-41cf-ba86-7676169148f5","Type":"ContainerStarted","Data":"c671c89cc603f423bcadfe87ced9b02577c8a096711674ba649d3451ffe67b27"}
Sep 16 17:35:18.620967 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:18.620942    2577 generic.go:356] "Generic (PLEG): container finished" podID="50d32d47-f631-4886-84bd-309cb9dfaccb" containerID="1c1dfcf7d53251a34b49005e4bb3a4acabbd56f4b1c0fb16a88aa7151370ac14" exitCode=0
Sep 16 17:35:18.621221 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:18.621024    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8vpf" event={"ID":"50d32d47-f631-4886-84bd-309cb9dfaccb","Type":"ContainerDied","Data":"1c1dfcf7d53251a34b49005e4bb3a4acabbd56f4b1c0fb16a88aa7151370ac14"}
Sep 16 17:35:18.621221 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:18.621068    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8vpf" event={"ID":"50d32d47-f631-4886-84bd-309cb9dfaccb","Type":"ContainerStarted","Data":"c5e99b11b5ea1d9eb0c2d355cf889ac2702822cde24967df6d51c9329a396360"}
Sep 16 17:35:18.622682 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:18.622665    2577 generic.go:356] "Generic (PLEG): container finished" podID="aeb7f372-1946-49ec-8599-d4db22b63d96" containerID="8a9b2b8b0a44c3e934f8b63262b0b43929d23e06ba01ea5728e071d4466ec620" exitCode=0
Sep 16 17:35:18.622772 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:18.622748    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9ssg" event={"ID":"aeb7f372-1946-49ec-8599-d4db22b63d96","Type":"ContainerDied","Data":"8a9b2b8b0a44c3e934f8b63262b0b43929d23e06ba01ea5728e071d4466ec620"}
Sep 16 17:35:18.622843 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:18.622792    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9ssg" event={"ID":"aeb7f372-1946-49ec-8599-d4db22b63d96","Type":"ContainerStarted","Data":"c4ef219b836f8c8b6755cf3c571cdd4ae85ee1902a904988ab424b82c0c5bb1d"}
Sep 16 17:35:18.624437 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:18.624419    2577 generic.go:356] "Generic (PLEG): container finished" podID="47a51c35-472e-41a7-a21b-67f12670cc8c" containerID="281ae41d976bd101de4683a3a7ea2f5d6478cd1ac93e76a87766d7390ba622b5" exitCode=0
Sep 16 17:35:18.624510 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:18.624443    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qp4gp" event={"ID":"47a51c35-472e-41a7-a21b-67f12670cc8c","Type":"ContainerDied","Data":"281ae41d976bd101de4683a3a7ea2f5d6478cd1ac93e76a87766d7390ba622b5"}
Sep 16 17:35:18.624510 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:18.624468    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qp4gp" event={"ID":"47a51c35-472e-41a7-a21b-67f12670cc8c","Type":"ContainerStarted","Data":"6bc039e7d056fe6e52b73730105b3329196dcff4e813a876e1539c9b18ef7657"}
Sep 16 17:35:19.632717 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:19.632680    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8vpf" event={"ID":"50d32d47-f631-4886-84bd-309cb9dfaccb","Type":"ContainerStarted","Data":"a6df424e954db0af397bed82cd42b21df6e6eed03ea598ec6dda9be26ce8498d"}
Sep 16 17:35:20.639128 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:20.639086    2577 generic.go:356] "Generic (PLEG): container finished" podID="47a51c35-472e-41a7-a21b-67f12670cc8c" containerID="8405a9b2d41c7c91ebac1a8e2f56d787c8df3f49bf1b142af74ee5270b6eefb3" exitCode=0
Sep 16 17:35:20.639568 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:20.639176    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qp4gp" event={"ID":"47a51c35-472e-41a7-a21b-67f12670cc8c","Type":"ContainerDied","Data":"8405a9b2d41c7c91ebac1a8e2f56d787c8df3f49bf1b142af74ee5270b6eefb3"}
Sep 16 17:35:20.641104 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:20.641086    2577 generic.go:356] "Generic (PLEG): container finished" podID="86fd1af6-b313-41cf-ba86-7676169148f5" containerID="647118489a78e6e2ddddba8ade7210c88013e678853171c9769ac81181b7e6f9" exitCode=0
Sep 16 17:35:20.641178 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:20.641152    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcxq4" event={"ID":"86fd1af6-b313-41cf-ba86-7676169148f5","Type":"ContainerDied","Data":"647118489a78e6e2ddddba8ade7210c88013e678853171c9769ac81181b7e6f9"}
Sep 16 17:35:20.643117 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:20.643088    2577 generic.go:356] "Generic (PLEG): container finished" podID="50d32d47-f631-4886-84bd-309cb9dfaccb" containerID="a6df424e954db0af397bed82cd42b21df6e6eed03ea598ec6dda9be26ce8498d" exitCode=0
Sep 16 17:35:20.643209 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:20.643166    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8vpf" event={"ID":"50d32d47-f631-4886-84bd-309cb9dfaccb","Type":"ContainerDied","Data":"a6df424e954db0af397bed82cd42b21df6e6eed03ea598ec6dda9be26ce8498d"}
Sep 16 17:35:21.649234 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:21.649202    2577 generic.go:356] "Generic (PLEG): container finished" podID="47a51c35-472e-41a7-a21b-67f12670cc8c" containerID="6b23b98477ba660da27ddfd6685ee3b5f26172d369979ee5cd8d98e1304aaf30" exitCode=0
Sep 16 17:35:21.649678 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:21.649291    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qp4gp" event={"ID":"47a51c35-472e-41a7-a21b-67f12670cc8c","Type":"ContainerDied","Data":"6b23b98477ba660da27ddfd6685ee3b5f26172d369979ee5cd8d98e1304aaf30"}
Sep 16 17:35:21.651373 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:21.651354    2577 generic.go:356] "Generic (PLEG): container finished" podID="86fd1af6-b313-41cf-ba86-7676169148f5" containerID="716d7e50275ced3074a91c0c008d241ec2599be0e479d6593135f716f0d9e24c" exitCode=0
Sep 16 17:35:21.651476 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:21.651416    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcxq4" event={"ID":"86fd1af6-b313-41cf-ba86-7676169148f5","Type":"ContainerDied","Data":"716d7e50275ced3074a91c0c008d241ec2599be0e479d6593135f716f0d9e24c"}
Sep 16 17:35:21.653505 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:21.653485    2577 generic.go:356] "Generic (PLEG): container finished" podID="50d32d47-f631-4886-84bd-309cb9dfaccb" containerID="9f4ca863dbd5932d575aef340adde9ecf2dab91944b18f837c0dfd6600c48b77" exitCode=0
Sep 16 17:35:21.653606 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:21.653579    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8vpf" event={"ID":"50d32d47-f631-4886-84bd-309cb9dfaccb","Type":"ContainerDied","Data":"9f4ca863dbd5932d575aef340adde9ecf2dab91944b18f837c0dfd6600c48b77"}
Sep 16 17:35:22.833740 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:22.833710    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8vpf"
Sep 16 17:35:22.836714 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:22.836693    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcxq4"
Sep 16 17:35:22.840125 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:22.840102    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qp4gp"
Sep 16 17:35:22.889552 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:22.889521    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50d32d47-f631-4886-84bd-309cb9dfaccb-bundle\") pod \"50d32d47-f631-4886-84bd-309cb9dfaccb\" (UID: \"50d32d47-f631-4886-84bd-309cb9dfaccb\") "
Sep 16 17:35:22.889728 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:22.889596    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50d32d47-f631-4886-84bd-309cb9dfaccb-util\") pod \"50d32d47-f631-4886-84bd-309cb9dfaccb\" (UID: \"50d32d47-f631-4886-84bd-309cb9dfaccb\") "
Sep 16 17:35:22.889728 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:22.889626    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47a51c35-472e-41a7-a21b-67f12670cc8c-util\") pod \"47a51c35-472e-41a7-a21b-67f12670cc8c\" (UID: \"47a51c35-472e-41a7-a21b-67f12670cc8c\") "
Sep 16 17:35:22.889728 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:22.889669    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86fd1af6-b313-41cf-ba86-7676169148f5-util\") pod \"86fd1af6-b313-41cf-ba86-7676169148f5\" (UID: \"86fd1af6-b313-41cf-ba86-7676169148f5\") "
Sep 16 17:35:22.889728 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:22.889709    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9s9cr\" (UniqueName: \"kubernetes.io/projected/86fd1af6-b313-41cf-ba86-7676169148f5-kube-api-access-9s9cr\") pod \"86fd1af6-b313-41cf-ba86-7676169148f5\" (UID: \"86fd1af6-b313-41cf-ba86-7676169148f5\") "
Sep 16 17:35:22.889961 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:22.889735    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86fd1af6-b313-41cf-ba86-7676169148f5-bundle\") pod \"86fd1af6-b313-41cf-ba86-7676169148f5\" (UID: \"86fd1af6-b313-41cf-ba86-7676169148f5\") "
Sep 16 17:35:22.889961 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:22.889759    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47a51c35-472e-41a7-a21b-67f12670cc8c-bundle\") pod \"47a51c35-472e-41a7-a21b-67f12670cc8c\" (UID: \"47a51c35-472e-41a7-a21b-67f12670cc8c\") "
Sep 16 17:35:22.889961 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:22.889789    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shqbx\" (UniqueName: \"kubernetes.io/projected/47a51c35-472e-41a7-a21b-67f12670cc8c-kube-api-access-shqbx\") pod \"47a51c35-472e-41a7-a21b-67f12670cc8c\" (UID: \"47a51c35-472e-41a7-a21b-67f12670cc8c\") "
Sep 16 17:35:22.889961 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:22.889838    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vjgf\" (UniqueName: \"kubernetes.io/projected/50d32d47-f631-4886-84bd-309cb9dfaccb-kube-api-access-5vjgf\") pod \"50d32d47-f631-4886-84bd-309cb9dfaccb\" (UID: \"50d32d47-f631-4886-84bd-309cb9dfaccb\") "
Sep 16 17:35:22.890912 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:22.890280    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a51c35-472e-41a7-a21b-67f12670cc8c-bundle" (OuterVolumeSpecName: "bundle") pod "47a51c35-472e-41a7-a21b-67f12670cc8c" (UID: "47a51c35-472e-41a7-a21b-67f12670cc8c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:35:22.890912 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:22.890560    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86fd1af6-b313-41cf-ba86-7676169148f5-bundle" (OuterVolumeSpecName: "bundle") pod "86fd1af6-b313-41cf-ba86-7676169148f5" (UID: "86fd1af6-b313-41cf-ba86-7676169148f5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:35:22.891127 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:22.891096    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50d32d47-f631-4886-84bd-309cb9dfaccb-bundle" (OuterVolumeSpecName: "bundle") pod "50d32d47-f631-4886-84bd-309cb9dfaccb" (UID: "50d32d47-f631-4886-84bd-309cb9dfaccb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:35:22.892223 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:22.892195    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50d32d47-f631-4886-84bd-309cb9dfaccb-kube-api-access-5vjgf" (OuterVolumeSpecName: "kube-api-access-5vjgf") pod "50d32d47-f631-4886-84bd-309cb9dfaccb" (UID: "50d32d47-f631-4886-84bd-309cb9dfaccb"). InnerVolumeSpecName "kube-api-access-5vjgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:35:22.892453 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:22.892434    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86fd1af6-b313-41cf-ba86-7676169148f5-kube-api-access-9s9cr" (OuterVolumeSpecName: "kube-api-access-9s9cr") pod "86fd1af6-b313-41cf-ba86-7676169148f5" (UID: "86fd1af6-b313-41cf-ba86-7676169148f5"). InnerVolumeSpecName "kube-api-access-9s9cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:35:22.892815 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:22.892798    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47a51c35-472e-41a7-a21b-67f12670cc8c-kube-api-access-shqbx" (OuterVolumeSpecName: "kube-api-access-shqbx") pod "47a51c35-472e-41a7-a21b-67f12670cc8c" (UID: "47a51c35-472e-41a7-a21b-67f12670cc8c"). InnerVolumeSpecName "kube-api-access-shqbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:35:22.897448 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:22.897421    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86fd1af6-b313-41cf-ba86-7676169148f5-util" (OuterVolumeSpecName: "util") pod "86fd1af6-b313-41cf-ba86-7676169148f5" (UID: "86fd1af6-b313-41cf-ba86-7676169148f5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:35:22.897852 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:22.897829    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47a51c35-472e-41a7-a21b-67f12670cc8c-util" (OuterVolumeSpecName: "util") pod "47a51c35-472e-41a7-a21b-67f12670cc8c" (UID: "47a51c35-472e-41a7-a21b-67f12670cc8c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:35:22.898482 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:22.898465    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50d32d47-f631-4886-84bd-309cb9dfaccb-util" (OuterVolumeSpecName: "util") pod "50d32d47-f631-4886-84bd-309cb9dfaccb" (UID: "50d32d47-f631-4886-84bd-309cb9dfaccb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:35:22.990818 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:22.990717    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47a51c35-472e-41a7-a21b-67f12670cc8c-util\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:35:22.990818 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:22.990757    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86fd1af6-b313-41cf-ba86-7676169148f5-util\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:35:22.990818 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:22.990771    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s9cr\" (UniqueName: \"kubernetes.io/projected/86fd1af6-b313-41cf-ba86-7676169148f5-kube-api-access-9s9cr\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:35:22.990818 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:22.990784    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86fd1af6-b313-41cf-ba86-7676169148f5-bundle\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:35:22.990818 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:22.990797    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47a51c35-472e-41a7-a21b-67f12670cc8c-bundle\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:35:22.990818 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:22.990808    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shqbx\" (UniqueName: \"kubernetes.io/projected/47a51c35-472e-41a7-a21b-67f12670cc8c-kube-api-access-shqbx\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:35:22.990818 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:22.990820    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vjgf\" (UniqueName: \"kubernetes.io/projected/50d32d47-f631-4886-84bd-309cb9dfaccb-kube-api-access-5vjgf\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:35:22.991163 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:22.990831    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50d32d47-f631-4886-84bd-309cb9dfaccb-bundle\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:35:22.991163 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:22.990843    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50d32d47-f631-4886-84bd-309cb9dfaccb-util\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:35:23.665725 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:23.665689    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qp4gp" event={"ID":"47a51c35-472e-41a7-a21b-67f12670cc8c","Type":"ContainerDied","Data":"6bc039e7d056fe6e52b73730105b3329196dcff4e813a876e1539c9b18ef7657"}
Sep 16 17:35:23.665918 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:23.665750    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bc039e7d056fe6e52b73730105b3329196dcff4e813a876e1539c9b18ef7657"
Sep 16 17:35:23.665918 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:23.665703    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qp4gp"
Sep 16 17:35:23.667675 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:23.667652    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcxq4"
Sep 16 17:35:23.667827 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:23.667647    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alcxq4" event={"ID":"86fd1af6-b313-41cf-ba86-7676169148f5","Type":"ContainerDied","Data":"c671c89cc603f423bcadfe87ced9b02577c8a096711674ba649d3451ffe67b27"}
Sep 16 17:35:23.667913 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:23.667855    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c671c89cc603f423bcadfe87ced9b02577c8a096711674ba649d3451ffe67b27"
Sep 16 17:35:23.669961 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:23.669941    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8vpf" event={"ID":"50d32d47-f631-4886-84bd-309cb9dfaccb","Type":"ContainerDied","Data":"c5e99b11b5ea1d9eb0c2d355cf889ac2702822cde24967df6d51c9329a396360"}
Sep 16 17:35:23.670041 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:23.669973    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5e99b11b5ea1d9eb0c2d355cf889ac2702822cde24967df6d51c9329a396360"
Sep 16 17:35:23.670041 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:23.670020    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v8vpf"
Sep 16 17:35:24.676762 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:24.676726    2577 generic.go:356] "Generic (PLEG): container finished" podID="aeb7f372-1946-49ec-8599-d4db22b63d96" containerID="4fa0631443f1ca5e0dc4ba92e7f7c4cbd6b7664d85c65290ab7d5d4df8e37cbf" exitCode=0
Sep 16 17:35:24.677169 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:24.676799    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9ssg" event={"ID":"aeb7f372-1946-49ec-8599-d4db22b63d96","Type":"ContainerDied","Data":"4fa0631443f1ca5e0dc4ba92e7f7c4cbd6b7664d85c65290ab7d5d4df8e37cbf"}
Sep 16 17:35:25.683880 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:25.683827    2577 generic.go:356] "Generic (PLEG): container finished" podID="aeb7f372-1946-49ec-8599-d4db22b63d96" containerID="015056e95d5a0752d6a8ee5e49d0d823401b20bfd526d55f74ad940191c4172c" exitCode=0
Sep 16 17:35:25.684259 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:25.683913    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9ssg" event={"ID":"aeb7f372-1946-49ec-8599-d4db22b63d96","Type":"ContainerDied","Data":"015056e95d5a0752d6a8ee5e49d0d823401b20bfd526d55f74ad940191c4172c"}
Sep 16 17:35:26.812004 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:26.811981    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9ssg"
Sep 16 17:35:26.922907 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:26.922852    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjvng\" (UniqueName: \"kubernetes.io/projected/aeb7f372-1946-49ec-8599-d4db22b63d96-kube-api-access-gjvng\") pod \"aeb7f372-1946-49ec-8599-d4db22b63d96\" (UID: \"aeb7f372-1946-49ec-8599-d4db22b63d96\") "
Sep 16 17:35:26.923077 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:26.922920    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aeb7f372-1946-49ec-8599-d4db22b63d96-util\") pod \"aeb7f372-1946-49ec-8599-d4db22b63d96\" (UID: \"aeb7f372-1946-49ec-8599-d4db22b63d96\") "
Sep 16 17:35:26.923077 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:26.923027    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aeb7f372-1946-49ec-8599-d4db22b63d96-bundle\") pod \"aeb7f372-1946-49ec-8599-d4db22b63d96\" (UID: \"aeb7f372-1946-49ec-8599-d4db22b63d96\") "
Sep 16 17:35:26.923494 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:26.923472    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeb7f372-1946-49ec-8599-d4db22b63d96-bundle" (OuterVolumeSpecName: "bundle") pod "aeb7f372-1946-49ec-8599-d4db22b63d96" (UID: "aeb7f372-1946-49ec-8599-d4db22b63d96"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:35:26.924957 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:26.924935    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeb7f372-1946-49ec-8599-d4db22b63d96-kube-api-access-gjvng" (OuterVolumeSpecName: "kube-api-access-gjvng") pod "aeb7f372-1946-49ec-8599-d4db22b63d96" (UID: "aeb7f372-1946-49ec-8599-d4db22b63d96"). InnerVolumeSpecName "kube-api-access-gjvng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:35:26.927588 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:26.927564    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeb7f372-1946-49ec-8599-d4db22b63d96-util" (OuterVolumeSpecName: "util") pod "aeb7f372-1946-49ec-8599-d4db22b63d96" (UID: "aeb7f372-1946-49ec-8599-d4db22b63d96"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:35:27.024489 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:27.024414    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aeb7f372-1946-49ec-8599-d4db22b63d96-bundle\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:35:27.024489 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:27.024440    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjvng\" (UniqueName: \"kubernetes.io/projected/aeb7f372-1946-49ec-8599-d4db22b63d96-kube-api-access-gjvng\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:35:27.024489 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:27.024449    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aeb7f372-1946-49ec-8599-d4db22b63d96-util\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:35:27.696095 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:27.696072    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9ssg"
Sep 16 17:35:27.696263 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:27.696097    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf9ssg" event={"ID":"aeb7f372-1946-49ec-8599-d4db22b63d96","Type":"ContainerDied","Data":"c4ef219b836f8c8b6755cf3c571cdd4ae85ee1902a904988ab424b82c0c5bb1d"}
Sep 16 17:35:27.696263 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:27.696139    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4ef219b836f8c8b6755cf3c571cdd4ae85ee1902a904988ab424b82c0c5bb1d"
Sep 16 17:35:32.797693 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:32.797657    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:34.644473 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.644437    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-45f5c"]
Sep 16 17:35:34.644849 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.644715    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50d32d47-f631-4886-84bd-309cb9dfaccb" containerName="pull"
Sep 16 17:35:34.644849 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.644726    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="50d32d47-f631-4886-84bd-309cb9dfaccb" containerName="pull"
Sep 16 17:35:34.644849 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.644732    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47a51c35-472e-41a7-a21b-67f12670cc8c" containerName="extract"
Sep 16 17:35:34.644849 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.644738    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="47a51c35-472e-41a7-a21b-67f12670cc8c" containerName="extract"
Sep 16 17:35:34.644849 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.644746    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeb7f372-1946-49ec-8599-d4db22b63d96" containerName="pull"
Sep 16 17:35:34.644849 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.644751    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeb7f372-1946-49ec-8599-d4db22b63d96" containerName="pull"
Sep 16 17:35:34.644849 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.644759    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50d32d47-f631-4886-84bd-309cb9dfaccb" containerName="util"
Sep 16 17:35:34.644849 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.644764    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="50d32d47-f631-4886-84bd-309cb9dfaccb" containerName="util"
Sep 16 17:35:34.644849 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.644773    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86fd1af6-b313-41cf-ba86-7676169148f5" containerName="pull"
Sep 16 17:35:34.644849 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.644779    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="86fd1af6-b313-41cf-ba86-7676169148f5" containerName="pull"
Sep 16 17:35:34.644849 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.644789    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86fd1af6-b313-41cf-ba86-7676169148f5" containerName="extract"
Sep 16 17:35:34.644849 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.644794    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="86fd1af6-b313-41cf-ba86-7676169148f5" containerName="extract"
Sep 16 17:35:34.644849 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.644800    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeb7f372-1946-49ec-8599-d4db22b63d96" containerName="extract"
Sep 16 17:35:34.644849 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.644804    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeb7f372-1946-49ec-8599-d4db22b63d96" containerName="extract"
Sep 16 17:35:34.644849 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.644812    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47a51c35-472e-41a7-a21b-67f12670cc8c" containerName="util"
Sep 16 17:35:34.644849 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.644817    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="47a51c35-472e-41a7-a21b-67f12670cc8c" containerName="util"
Sep 16 17:35:34.644849 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.644822    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86fd1af6-b313-41cf-ba86-7676169148f5" containerName="util"
Sep 16 17:35:34.644849 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.644827    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="86fd1af6-b313-41cf-ba86-7676169148f5" containerName="util"
Sep 16 17:35:34.644849 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.644834    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeb7f372-1946-49ec-8599-d4db22b63d96" containerName="util"
Sep 16 17:35:34.644849 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.644839    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeb7f372-1946-49ec-8599-d4db22b63d96" containerName="util"
Sep 16 17:35:34.644849 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.644845    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50d32d47-f631-4886-84bd-309cb9dfaccb" containerName="extract"
Sep 16 17:35:34.644849 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.644849    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="50d32d47-f631-4886-84bd-309cb9dfaccb" containerName="extract"
Sep 16 17:35:34.644849 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.644856    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47a51c35-472e-41a7-a21b-67f12670cc8c" containerName="pull"
Sep 16 17:35:34.644849 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.644877    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="47a51c35-472e-41a7-a21b-67f12670cc8c" containerName="pull"
Sep 16 17:35:34.645524 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.644927    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="86fd1af6-b313-41cf-ba86-7676169148f5" containerName="extract"
Sep 16 17:35:34.645524 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.644935    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeb7f372-1946-49ec-8599-d4db22b63d96" containerName="extract"
Sep 16 17:35:34.645524 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.644942    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="50d32d47-f631-4886-84bd-309cb9dfaccb" containerName="extract"
Sep 16 17:35:34.645524 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.644949    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="47a51c35-472e-41a7-a21b-67f12670cc8c" containerName="extract"
Sep 16 17:35:34.647709 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.647692    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-45f5c"
Sep 16 17:35:34.651570 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.651550    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 17:35:34.651686 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.651607    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 17:35:34.651686 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.651679    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-vl294\""
Sep 16 17:35:34.652557 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.652542    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 16 17:35:34.661328 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.661304    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-45f5c"]
Sep 16 17:35:34.686983 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.686961    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfw4p\" (UniqueName: \"kubernetes.io/projected/191cce97-72b1-40a1-a139-5a48ab187aa2-kube-api-access-qfw4p\") pod \"dns-operator-controller-manager-65dc8f76f5-45f5c\" (UID: \"191cce97-72b1-40a1-a139-5a48ab187aa2\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-45f5c"
Sep 16 17:35:34.788054 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.788009    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfw4p\" (UniqueName: \"kubernetes.io/projected/191cce97-72b1-40a1-a139-5a48ab187aa2-kube-api-access-qfw4p\") pod \"dns-operator-controller-manager-65dc8f76f5-45f5c\" (UID: \"191cce97-72b1-40a1-a139-5a48ab187aa2\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-45f5c"
Sep 16 17:35:34.798500 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.798477    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfw4p\" (UniqueName: \"kubernetes.io/projected/191cce97-72b1-40a1-a139-5a48ab187aa2-kube-api-access-qfw4p\") pod \"dns-operator-controller-manager-65dc8f76f5-45f5c\" (UID: \"191cce97-72b1-40a1-a139-5a48ab187aa2\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-45f5c"
Sep 16 17:35:34.958084 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:34.957987    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-45f5c"
Sep 16 17:35:35.232105 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:35.232074    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-45f5c"]
Sep 16 17:35:35.233832 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:35:35.233798    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod191cce97_72b1_40a1_a139_5a48ab187aa2.slice/crio-b472547b97af20e2554e4d1a6e83c8d60ae54576bd62de06dbebaeff01266242 WatchSource:0}: Error finding container b472547b97af20e2554e4d1a6e83c8d60ae54576bd62de06dbebaeff01266242: Status 404 returned error can't find the container with id b472547b97af20e2554e4d1a6e83c8d60ae54576bd62de06dbebaeff01266242
Sep 16 17:35:35.375058 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:35.375019    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:35.736263 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:35.736225    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-45f5c" event={"ID":"191cce97-72b1-40a1-a139-5a48ab187aa2","Type":"ContainerStarted","Data":"b472547b97af20e2554e4d1a6e83c8d60ae54576bd62de06dbebaeff01266242"}
Sep 16 17:35:36.655901 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:36.655846    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-vd7dc"]
Sep 16 17:35:36.660274 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:36.660250    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-vd7dc"
Sep 16 17:35:36.663385 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:36.663363    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-mbxwr\""
Sep 16 17:35:36.673613 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:36.673590    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-vd7dc"]
Sep 16 17:35:36.705135 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:36.705098    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2hvn\" (UniqueName: \"kubernetes.io/projected/0a064f1b-b82e-4155-ba16-58fac8703bcd-kube-api-access-g2hvn\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-vd7dc\" (UID: \"0a064f1b-b82e-4155-ba16-58fac8703bcd\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-vd7dc"
Sep 16 17:35:36.806083 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:36.806036    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2hvn\" (UniqueName: \"kubernetes.io/projected/0a064f1b-b82e-4155-ba16-58fac8703bcd-kube-api-access-g2hvn\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-vd7dc\" (UID: \"0a064f1b-b82e-4155-ba16-58fac8703bcd\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-vd7dc"
Sep 16 17:35:36.822921 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:36.822895    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2hvn\" (UniqueName: \"kubernetes.io/projected/0a064f1b-b82e-4155-ba16-58fac8703bcd-kube-api-access-g2hvn\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-vd7dc\" (UID: \"0a064f1b-b82e-4155-ba16-58fac8703bcd\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-vd7dc"
Sep 16 17:35:36.970550 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:36.970451    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-vd7dc"
Sep 16 17:35:37.093212 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:37.093178    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:37.186580 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:37.186540    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:37.251357 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:37.251333    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-vd7dc"]
Sep 16 17:35:37.253160 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:35:37.253125    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a064f1b_b82e_4155_ba16_58fac8703bcd.slice/crio-b5018a1f30a02745f96fa1f2e91168e4430f0c12d25c13020b916d45ffc90cd2 WatchSource:0}: Error finding container b5018a1f30a02745f96fa1f2e91168e4430f0c12d25c13020b916d45ffc90cd2: Status 404 returned error can't find the container with id b5018a1f30a02745f96fa1f2e91168e4430f0c12d25c13020b916d45ffc90cd2
Sep 16 17:35:37.748170 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:37.748133    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-vd7dc" event={"ID":"0a064f1b-b82e-4155-ba16-58fac8703bcd","Type":"ContainerStarted","Data":"b5018a1f30a02745f96fa1f2e91168e4430f0c12d25c13020b916d45ffc90cd2"}
Sep 16 17:35:37.771614 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:37.771583    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:38.184745 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:38.184706    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:38.602852 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:38.602809    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:39.259231 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:39.259139    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:39.405651 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:39.405600    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:39.760372 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:39.760331    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-vd7dc" event={"ID":"0a064f1b-b82e-4155-ba16-58fac8703bcd","Type":"ContainerStarted","Data":"5613c6e58131a1eeb802aca92116fe70288ecd58d04ed41cb64cf8e08020a6b4"}
Sep 16 17:35:39.760566 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:39.760455    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-vd7dc"
Sep 16 17:35:39.782775 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:39.782717    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-vd7dc" podStartSLOduration=2.242169407 podStartE2EDuration="3.782696518s" totalImagesPullingTime="1.540527111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:35:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:35:39.780308376 +0000 UTC m=+661.142987186" watchObservedRunningTime="2026-09-16 17:35:39.782696518 +0000 UTC m=+661.145375307"
Sep 16 17:35:41.634702 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:41.634660    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:41.914605 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:41.914579    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 16 17:35:41.983566 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:41.983540    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:42.778173 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:42.778136    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-45f5c" event={"ID":"191cce97-72b1-40a1-a139-5a48ab187aa2","Type":"ContainerStarted","Data":"11de9c0e1768e2b0b441077e71dde2fc4440f1692a99af0815e3acd5f82947b7"}
Sep 16 17:35:42.778623 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:42.778302    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-45f5c"
Sep 16 17:35:42.806408 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:42.806351    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-45f5c" podStartSLOduration=2.131720287 podStartE2EDuration="8.806336766s" totalImagesPullingTime="6.674616479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:35:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:35:42.804829388 +0000 UTC m=+664.167508174" watchObservedRunningTime="2026-09-16 17:35:42.806336766 +0000 UTC m=+664.169015552"
Sep 16 17:35:43.054967 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:43.054853    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:44.304921 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:44.304881    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-drm6x"]
Sep 16 17:35:44.308329 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:44.308308    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-drm6x"
Sep 16 17:35:44.311124 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:44.311105    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-6w4sz\""
Sep 16 17:35:44.328405 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:44.328375    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-drm6x"]
Sep 16 17:35:44.371694 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:44.371663    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5dvn\" (UniqueName: \"kubernetes.io/projected/7f849eb8-5b46-48b5-b313-f2fd773c467e-kube-api-access-c5dvn\") pod \"authorino-operator-6fc8f864f4-drm6x\" (UID: \"7f849eb8-5b46-48b5-b313-f2fd773c467e\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-drm6x"
Sep 16 17:35:44.472593 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:44.472553    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5dvn\" (UniqueName: \"kubernetes.io/projected/7f849eb8-5b46-48b5-b313-f2fd773c467e-kube-api-access-c5dvn\") pod \"authorino-operator-6fc8f864f4-drm6x\" (UID: \"7f849eb8-5b46-48b5-b313-f2fd773c467e\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-drm6x"
Sep 16 17:35:44.486702 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:44.486670    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5dvn\" (UniqueName: \"kubernetes.io/projected/7f849eb8-5b46-48b5-b313-f2fd773c467e-kube-api-access-c5dvn\") pod \"authorino-operator-6fc8f864f4-drm6x\" (UID: \"7f849eb8-5b46-48b5-b313-f2fd773c467e\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-drm6x"
Sep 16 17:35:44.617962 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:44.617927    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-drm6x"
Sep 16 17:35:44.905480 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:44.905432    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-drm6x"]
Sep 16 17:35:44.907659 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:35:44.907629    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f849eb8_5b46_48b5_b313_f2fd773c467e.slice/crio-365cd0687249e40c1d7222fe0e72ef2ee94f4953c0487b811b5c2bdb34d1c01d WatchSource:0}: Error finding container 365cd0687249e40c1d7222fe0e72ef2ee94f4953c0487b811b5c2bdb34d1c01d: Status 404 returned error can't find the container with id 365cd0687249e40c1d7222fe0e72ef2ee94f4953c0487b811b5c2bdb34d1c01d
Sep 16 17:35:45.453025 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:45.452986    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:45.556167 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:45.556129    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:45.793625 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:45.793529    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-drm6x" event={"ID":"7f849eb8-5b46-48b5-b313-f2fd773c467e","Type":"ContainerStarted","Data":"365cd0687249e40c1d7222fe0e72ef2ee94f4953c0487b811b5c2bdb34d1c01d"}
Sep 16 17:35:47.805748 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:47.805709    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-drm6x" event={"ID":"7f849eb8-5b46-48b5-b313-f2fd773c467e","Type":"ContainerStarted","Data":"1bf09764d027a456b14083d22aa2b3853a33c43370aa6652cd0203f7de8bca01"}
Sep 16 17:35:47.806144 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:47.805808    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-drm6x"
Sep 16 17:35:47.831140 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:47.831088    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6fc8f864f4-drm6x" podStartSLOduration=1.8115085720000002 podStartE2EDuration="3.831072721s" totalImagesPullingTime="2.019564149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:35:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:35:47.828279484 +0000 UTC m=+669.190958269" watchObservedRunningTime="2026-09-16 17:35:47.831072721 +0000 UTC m=+669.193751506"
Sep 16 17:35:49.002527 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:49.002494    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:49.100281 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:49.100248    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:50.253522 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:50.253493    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:50.476028 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:50.475997    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:50.767255 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:50.767228    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-vd7dc"
Sep 16 17:35:53.168377 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:53.168342    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:53.288788 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:53.288755    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:53.416738 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:53.416703    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:35:53.784997 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:53.784970    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-45f5c"
Sep 16 17:35:58.813759 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:35:58.813719    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-drm6x"
Sep 16 17:36:01.956754 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:01.956702    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:36:02.058448 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:02.058410    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:36:02.160073 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:02.160039    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:36:04.537227 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:04.537192    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:36:04.641377 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:04.641330    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:36:07.767221 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:07.767192    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:36:07.918022 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:07.917978    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:36:12.386493 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:12.386452    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:36:12.648101 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:12.647982    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:36:12.945583 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:12.945472    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:36:21.057373 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:21.057328    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:36:21.160846 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:21.160807    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:36:27.529416 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:27.529374    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:36:27.617040 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:27.617001    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:36:33.732241 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:33.732152    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:36:33.797358 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:33.797315    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:36:34.526982 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:34.526942    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:36:34.634038 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:34.634001    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:36:34.710598 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:34.710563    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:36:35.010721 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:35.010686    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vd9tf"]
Sep 16 17:36:35.014401 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:35.014342    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-vd9tf"
Sep 16 17:36:35.017845 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:35.017825    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 16 17:36:35.018884 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:35.018848    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2xswh\""
Sep 16 17:36:35.029878 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:35.029836    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vd9tf"]
Sep 16 17:36:35.094716 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:35.094687    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d585b\" (UniqueName: \"kubernetes.io/projected/3b3e0345-ec43-4b4e-ba85-73f7cbb3e7d1-kube-api-access-d585b\") pod \"limitador-limitador-84994f7ffc-vd9tf\" (UID: \"3b3e0345-ec43-4b4e-ba85-73f7cbb3e7d1\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-vd9tf"
Sep 16 17:36:35.094899 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:35.094738    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3b3e0345-ec43-4b4e-ba85-73f7cbb3e7d1-config-file\") pod \"limitador-limitador-84994f7ffc-vd9tf\" (UID: \"3b3e0345-ec43-4b4e-ba85-73f7cbb3e7d1\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-vd9tf"
Sep 16 17:36:35.114271 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:35.114242    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vd9tf"]
Sep 16 17:36:35.195536 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:35.195499    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3b3e0345-ec43-4b4e-ba85-73f7cbb3e7d1-config-file\") pod \"limitador-limitador-84994f7ffc-vd9tf\" (UID: \"3b3e0345-ec43-4b4e-ba85-73f7cbb3e7d1\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-vd9tf"
Sep 16 17:36:35.195703 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:35.195581    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d585b\" (UniqueName: \"kubernetes.io/projected/3b3e0345-ec43-4b4e-ba85-73f7cbb3e7d1-kube-api-access-d585b\") pod \"limitador-limitador-84994f7ffc-vd9tf\" (UID: \"3b3e0345-ec43-4b4e-ba85-73f7cbb3e7d1\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-vd9tf"
Sep 16 17:36:35.196165 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:35.196146    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3b3e0345-ec43-4b4e-ba85-73f7cbb3e7d1-config-file\") pod \"limitador-limitador-84994f7ffc-vd9tf\" (UID: \"3b3e0345-ec43-4b4e-ba85-73f7cbb3e7d1\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-vd9tf"
Sep 16 17:36:35.210283 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:35.210259    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d585b\" (UniqueName: \"kubernetes.io/projected/3b3e0345-ec43-4b4e-ba85-73f7cbb3e7d1-kube-api-access-d585b\") pod \"limitador-limitador-84994f7ffc-vd9tf\" (UID: \"3b3e0345-ec43-4b4e-ba85-73f7cbb3e7d1\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-vd9tf"
Sep 16 17:36:35.326166 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:35.326128    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-vd9tf"
Sep 16 17:36:35.464794 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:35.464758    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:36:35.564675 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:35.564641    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:36:35.603744 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:35.603716    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vd9tf"]
Sep 16 17:36:35.604086 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:36:35.604063    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b3e0345_ec43_4b4e_ba85_73f7cbb3e7d1.slice/crio-6cef09ebe5da0a5f5452c474ccb875a08f086ffdf9980b512bad972fce323585 WatchSource:0}: Error finding container 6cef09ebe5da0a5f5452c474ccb875a08f086ffdf9980b512bad972fce323585: Status 404 returned error can't find the container with id 6cef09ebe5da0a5f5452c474ccb875a08f086ffdf9980b512bad972fce323585
Sep 16 17:36:36.048364 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:36.048280    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-vd9tf" event={"ID":"3b3e0345-ec43-4b4e-ba85-73f7cbb3e7d1","Type":"ContainerStarted","Data":"6cef09ebe5da0a5f5452c474ccb875a08f086ffdf9980b512bad972fce323585"}
Sep 16 17:36:37.055248 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:37.055217    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-vd9tf" event={"ID":"3b3e0345-ec43-4b4e-ba85-73f7cbb3e7d1","Type":"ContainerStarted","Data":"2236f1a2d1e722fcd92f445bf6228d5d86950b3e53bebd2cc674d5685f41f369"}
Sep 16 17:36:37.055651 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:37.055319    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-vd9tf"
Sep 16 17:36:37.077139 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:37.077088    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-vd9tf" podStartSLOduration=2.193190044 podStartE2EDuration="3.077067741s" totalImagesPullingTime="883.877697ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:36:37.075252026 +0000 UTC m=+718.437930814" watchObservedRunningTime="2026-09-16 17:36:37.077067741 +0000 UTC m=+718.439746528"
Sep 16 17:36:48.061521 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:48.061488    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-vd9tf"
Sep 16 17:36:54.224649 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:54.224610    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:36:54.407872 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:36:54.407836    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:37:02.772439 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:02.772398    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:37:02.873992 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:02.873952    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:37:23.710732 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:23.710694    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c"]
Sep 16 17:37:23.711199 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:23.710882    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c" podUID="d3482ef2-f205-428e-b2fa-0019c70fed06" containerName="discovery" containerID="cri-o://b372296129f74cf2fbd862cae8087eb626601068f4f9c95bd26f0ab7b078b92b" gracePeriod=30
Sep 16 17:37:23.953624 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:23.953597    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c"
Sep 16 17:37:23.995143 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:23.995070    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d3482ef2-f205-428e-b2fa-0019c70fed06-istio-kubeconfig\") pod \"d3482ef2-f205-428e-b2fa-0019c70fed06\" (UID: \"d3482ef2-f205-428e-b2fa-0019c70fed06\") "
Sep 16 17:37:23.995143 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:23.995108    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3482ef2-f205-428e-b2fa-0019c70fed06-istio-token\") pod \"d3482ef2-f205-428e-b2fa-0019c70fed06\" (UID: \"d3482ef2-f205-428e-b2fa-0019c70fed06\") "
Sep 16 17:37:23.995143 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:23.995127    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xd265\" (UniqueName: \"kubernetes.io/projected/d3482ef2-f205-428e-b2fa-0019c70fed06-kube-api-access-xd265\") pod \"d3482ef2-f205-428e-b2fa-0019c70fed06\" (UID: \"d3482ef2-f205-428e-b2fa-0019c70fed06\") "
Sep 16 17:37:23.995409 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:23.995168    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d3482ef2-f205-428e-b2fa-0019c70fed06-cacerts\") pod \"d3482ef2-f205-428e-b2fa-0019c70fed06\" (UID: \"d3482ef2-f205-428e-b2fa-0019c70fed06\") "
Sep 16 17:37:23.995409 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:23.995200    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d3482ef2-f205-428e-b2fa-0019c70fed06-istio-csr-ca-configmap\") pod \"d3482ef2-f205-428e-b2fa-0019c70fed06\" (UID: \"d3482ef2-f205-428e-b2fa-0019c70fed06\") "
Sep 16 17:37:23.995409 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:23.995221    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d3482ef2-f205-428e-b2fa-0019c70fed06-istio-csr-dns-cert\") pod \"d3482ef2-f205-428e-b2fa-0019c70fed06\" (UID: \"d3482ef2-f205-428e-b2fa-0019c70fed06\") "
Sep 16 17:37:23.995409 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:23.995265    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3482ef2-f205-428e-b2fa-0019c70fed06-local-certs\") pod \"d3482ef2-f205-428e-b2fa-0019c70fed06\" (UID: \"d3482ef2-f205-428e-b2fa-0019c70fed06\") "
Sep 16 17:37:23.995666 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:23.995637    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3482ef2-f205-428e-b2fa-0019c70fed06-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "d3482ef2-f205-428e-b2fa-0019c70fed06" (UID: "d3482ef2-f205-428e-b2fa-0019c70fed06"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:37:23.997749 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:23.997681    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3482ef2-f205-428e-b2fa-0019c70fed06-istio-token" (OuterVolumeSpecName: "istio-token") pod "d3482ef2-f205-428e-b2fa-0019c70fed06" (UID: "d3482ef2-f205-428e-b2fa-0019c70fed06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:37:23.997893 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:23.997803    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3482ef2-f205-428e-b2fa-0019c70fed06-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "d3482ef2-f205-428e-b2fa-0019c70fed06" (UID: "d3482ef2-f205-428e-b2fa-0019c70fed06"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:37:23.997893 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:23.997807    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3482ef2-f205-428e-b2fa-0019c70fed06-kube-api-access-xd265" (OuterVolumeSpecName: "kube-api-access-xd265") pod "d3482ef2-f205-428e-b2fa-0019c70fed06" (UID: "d3482ef2-f205-428e-b2fa-0019c70fed06"). InnerVolumeSpecName "kube-api-access-xd265". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:37:23.997893 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:23.997823    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3482ef2-f205-428e-b2fa-0019c70fed06-local-certs" (OuterVolumeSpecName: "local-certs") pod "d3482ef2-f205-428e-b2fa-0019c70fed06" (UID: "d3482ef2-f205-428e-b2fa-0019c70fed06"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:37:23.997893 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:23.997834    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3482ef2-f205-428e-b2fa-0019c70fed06-cacerts" (OuterVolumeSpecName: "cacerts") pod "d3482ef2-f205-428e-b2fa-0019c70fed06" (UID: "d3482ef2-f205-428e-b2fa-0019c70fed06"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:37:23.998161 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:23.997921    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3482ef2-f205-428e-b2fa-0019c70fed06-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "d3482ef2-f205-428e-b2fa-0019c70fed06" (UID: "d3482ef2-f205-428e-b2fa-0019c70fed06"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:37:24.096628 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:24.096590    2577 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d3482ef2-f205-428e-b2fa-0019c70fed06-istio-kubeconfig\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:37:24.096628 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:24.096619    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3482ef2-f205-428e-b2fa-0019c70fed06-istio-token\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:37:24.096628 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:24.096628    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd265\" (UniqueName: \"kubernetes.io/projected/d3482ef2-f205-428e-b2fa-0019c70fed06-kube-api-access-xd265\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:37:24.096628 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:24.096637    2577 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d3482ef2-f205-428e-b2fa-0019c70fed06-cacerts\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:37:24.096906 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:24.096646    2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d3482ef2-f205-428e-b2fa-0019c70fed06-istio-csr-ca-configmap\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:37:24.096906 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:24.096656    2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d3482ef2-f205-428e-b2fa-0019c70fed06-istio-csr-dns-cert\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:37:24.096906 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:24.096664    2577 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3482ef2-f205-428e-b2fa-0019c70fed06-local-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:37:24.299249 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:24.299156    2577 generic.go:356] "Generic (PLEG): container finished" podID="d3482ef2-f205-428e-b2fa-0019c70fed06" containerID="b372296129f74cf2fbd862cae8087eb626601068f4f9c95bd26f0ab7b078b92b" exitCode=0
Sep 16 17:37:24.299249 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:24.299207    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c"
Sep 16 17:37:24.299420 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:24.299249    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c" event={"ID":"d3482ef2-f205-428e-b2fa-0019c70fed06","Type":"ContainerDied","Data":"b372296129f74cf2fbd862cae8087eb626601068f4f9c95bd26f0ab7b078b92b"}
Sep 16 17:37:24.299420 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:24.299328    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c" event={"ID":"d3482ef2-f205-428e-b2fa-0019c70fed06","Type":"ContainerDied","Data":"5bf47333bbf62537fd4704cfc3e7bdb10ecb62f1cb38f775a45935de935a47b5"}
Sep 16 17:37:24.299420 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:24.299352    2577 scope.go:117] "RemoveContainer" containerID="b372296129f74cf2fbd862cae8087eb626601068f4f9c95bd26f0ab7b078b92b"
Sep 16 17:37:24.309033 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:24.309013    2577 scope.go:117] "RemoveContainer" containerID="b372296129f74cf2fbd862cae8087eb626601068f4f9c95bd26f0ab7b078b92b"
Sep 16 17:37:24.309307 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:37:24.309287    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b372296129f74cf2fbd862cae8087eb626601068f4f9c95bd26f0ab7b078b92b\": container with ID starting with b372296129f74cf2fbd862cae8087eb626601068f4f9c95bd26f0ab7b078b92b not found: ID does not exist" containerID="b372296129f74cf2fbd862cae8087eb626601068f4f9c95bd26f0ab7b078b92b"
Sep 16 17:37:24.309358 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:24.309316    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b372296129f74cf2fbd862cae8087eb626601068f4f9c95bd26f0ab7b078b92b"} err="failed to get container status \"b372296129f74cf2fbd862cae8087eb626601068f4f9c95bd26f0ab7b078b92b\": rpc error: code = NotFound desc = could not find container \"b372296129f74cf2fbd862cae8087eb626601068f4f9c95bd26f0ab7b078b92b\": container with ID starting with b372296129f74cf2fbd862cae8087eb626601068f4f9c95bd26f0ab7b078b92b not found: ID does not exist"
Sep 16 17:37:24.327799 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:24.327774    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c"]
Sep 16 17:37:24.329746 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:24.329723    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cp95c"]
Sep 16 17:37:25.080959 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:25.080931    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3482ef2-f205-428e-b2fa-0019c70fed06" path="/var/lib/kubelet/pods/d3482ef2-f205-428e-b2fa-0019c70fed06/volumes"
Sep 16 17:37:33.528100 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.528056    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:37:33.644833 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.644791    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:37:33.645842 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.645819    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-cc8594966-qs2df"]
Sep 16 17:37:33.646194 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.646181    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3482ef2-f205-428e-b2fa-0019c70fed06" containerName="discovery"
Sep 16 17:37:33.646243 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.646195    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3482ef2-f205-428e-b2fa-0019c70fed06" containerName="discovery"
Sep 16 17:37:33.646279 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.646245    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3482ef2-f205-428e-b2fa-0019c70fed06" containerName="discovery"
Sep 16 17:37:33.650418 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.650402    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cc8594966-qs2df"
Sep 16 17:37:33.654259 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.654237    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-d2v6x\""
Sep 16 17:37:33.654528 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.654514    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 17:37:33.654597 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.654516    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 16 17:37:33.654597 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.654553    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 17:37:33.666950 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.666931    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-cc8594966-qs2df"]
Sep 16 17:37:33.681228 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.681206    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-5f574bd55d-xlhdg"]
Sep 16 17:37:33.684360 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.684343    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-5f574bd55d-xlhdg"
Sep 16 17:37:33.688322 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.688303    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-rxcvx\""
Sep 16 17:37:33.688404 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.688305    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 16 17:37:33.688514 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.688496    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17b24012-a56c-45d8-ad67-a1c9d9d282e6-cert\") pod \"kserve-controller-manager-cc8594966-qs2df\" (UID: \"17b24012-a56c-45d8-ad67-a1c9d9d282e6\") " pod="kserve/kserve-controller-manager-cc8594966-qs2df"
Sep 16 17:37:33.688592 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.688574    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhsnb\" (UniqueName: \"kubernetes.io/projected/17b24012-a56c-45d8-ad67-a1c9d9d282e6-kube-api-access-dhsnb\") pod \"kserve-controller-manager-cc8594966-qs2df\" (UID: \"17b24012-a56c-45d8-ad67-a1c9d9d282e6\") " pod="kserve/kserve-controller-manager-cc8594966-qs2df"
Sep 16 17:37:33.702130 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.702109    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-5f574bd55d-xlhdg"]
Sep 16 17:37:33.756329 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.756294    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-6wwcz"]
Sep 16 17:37:33.759633 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.759610    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-6wwcz"
Sep 16 17:37:33.779932 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.779855    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 17:37:33.780506 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.780491    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-gqc2m\""
Sep 16 17:37:33.789399 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.789372    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhsnb\" (UniqueName: \"kubernetes.io/projected/17b24012-a56c-45d8-ad67-a1c9d9d282e6-kube-api-access-dhsnb\") pod \"kserve-controller-manager-cc8594966-qs2df\" (UID: \"17b24012-a56c-45d8-ad67-a1c9d9d282e6\") " pod="kserve/kserve-controller-manager-cc8594966-qs2df"
Sep 16 17:37:33.789521 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.789413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17b24012-a56c-45d8-ad67-a1c9d9d282e6-cert\") pod \"kserve-controller-manager-cc8594966-qs2df\" (UID: \"17b24012-a56c-45d8-ad67-a1c9d9d282e6\") " pod="kserve/kserve-controller-manager-cc8594966-qs2df"
Sep 16 17:37:33.789521 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.789465    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq42l\" (UniqueName: \"kubernetes.io/projected/8a0f5af5-641b-43af-823b-e001c9ec9e88-kube-api-access-kq42l\") pod \"llmisvc-controller-manager-5f574bd55d-xlhdg\" (UID: \"8a0f5af5-641b-43af-823b-e001c9ec9e88\") " pod="kserve/llmisvc-controller-manager-5f574bd55d-xlhdg"
Sep 16 17:37:33.789521 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.789497    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a0f5af5-641b-43af-823b-e001c9ec9e88-cert\") pod \"llmisvc-controller-manager-5f574bd55d-xlhdg\" (UID: \"8a0f5af5-641b-43af-823b-e001c9ec9e88\") " pod="kserve/llmisvc-controller-manager-5f574bd55d-xlhdg"
Sep 16 17:37:33.791835 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.791809    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17b24012-a56c-45d8-ad67-a1c9d9d282e6-cert\") pod \"kserve-controller-manager-cc8594966-qs2df\" (UID: \"17b24012-a56c-45d8-ad67-a1c9d9d282e6\") " pod="kserve/kserve-controller-manager-cc8594966-qs2df"
Sep 16 17:37:33.795637 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.795609    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:37:33.800412 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.800389    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-6wwcz"]
Sep 16 17:37:33.831427 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.831393    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhsnb\" (UniqueName: \"kubernetes.io/projected/17b24012-a56c-45d8-ad67-a1c9d9d282e6-kube-api-access-dhsnb\") pod \"kserve-controller-manager-cc8594966-qs2df\" (UID: \"17b24012-a56c-45d8-ad67-a1c9d9d282e6\") " pod="kserve/kserve-controller-manager-cc8594966-qs2df"
Sep 16 17:37:33.890255 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.890217    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a0f5af5-641b-43af-823b-e001c9ec9e88-cert\") pod \"llmisvc-controller-manager-5f574bd55d-xlhdg\" (UID: \"8a0f5af5-641b-43af-823b-e001c9ec9e88\") " pod="kserve/llmisvc-controller-manager-5f574bd55d-xlhdg"
Sep 16 17:37:33.890460 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.890273    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txsj5\" (UniqueName: \"kubernetes.io/projected/356b448f-a555-4ede-a47c-c4be3e8444c8-kube-api-access-txsj5\") pod \"seaweedfs-64568bcd49-6wwcz\" (UID: \"356b448f-a555-4ede-a47c-c4be3e8444c8\") " pod="kserve/seaweedfs-64568bcd49-6wwcz"
Sep 16 17:37:33.890460 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.890297    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/356b448f-a555-4ede-a47c-c4be3e8444c8-data\") pod \"seaweedfs-64568bcd49-6wwcz\" (UID: \"356b448f-a555-4ede-a47c-c4be3e8444c8\") " pod="kserve/seaweedfs-64568bcd49-6wwcz"
Sep 16 17:37:33.890460 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.890369    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq42l\" (UniqueName: \"kubernetes.io/projected/8a0f5af5-641b-43af-823b-e001c9ec9e88-kube-api-access-kq42l\") pod \"llmisvc-controller-manager-5f574bd55d-xlhdg\" (UID: \"8a0f5af5-641b-43af-823b-e001c9ec9e88\") " pod="kserve/llmisvc-controller-manager-5f574bd55d-xlhdg"
Sep 16 17:37:33.892560 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.892539    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a0f5af5-641b-43af-823b-e001c9ec9e88-cert\") pod \"llmisvc-controller-manager-5f574bd55d-xlhdg\" (UID: \"8a0f5af5-641b-43af-823b-e001c9ec9e88\") " pod="kserve/llmisvc-controller-manager-5f574bd55d-xlhdg"
Sep 16 17:37:33.906884 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.906843    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq42l\" (UniqueName: \"kubernetes.io/projected/8a0f5af5-641b-43af-823b-e001c9ec9e88-kube-api-access-kq42l\") pod \"llmisvc-controller-manager-5f574bd55d-xlhdg\" (UID: \"8a0f5af5-641b-43af-823b-e001c9ec9e88\") " pod="kserve/llmisvc-controller-manager-5f574bd55d-xlhdg"
Sep 16 17:37:33.960697 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.960662    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cc8594966-qs2df"
Sep 16 17:37:33.990981 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.990950    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txsj5\" (UniqueName: \"kubernetes.io/projected/356b448f-a555-4ede-a47c-c4be3e8444c8-kube-api-access-txsj5\") pod \"seaweedfs-64568bcd49-6wwcz\" (UID: \"356b448f-a555-4ede-a47c-c4be3e8444c8\") " pod="kserve/seaweedfs-64568bcd49-6wwcz"
Sep 16 17:37:33.991125 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.990993    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/356b448f-a555-4ede-a47c-c4be3e8444c8-data\") pod \"seaweedfs-64568bcd49-6wwcz\" (UID: \"356b448f-a555-4ede-a47c-c4be3e8444c8\") " pod="kserve/seaweedfs-64568bcd49-6wwcz"
Sep 16 17:37:33.991421 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.991397    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/356b448f-a555-4ede-a47c-c4be3e8444c8-data\") pod \"seaweedfs-64568bcd49-6wwcz\" (UID: \"356b448f-a555-4ede-a47c-c4be3e8444c8\") " pod="kserve/seaweedfs-64568bcd49-6wwcz"
Sep 16 17:37:33.993853 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:33.993834    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-5f574bd55d-xlhdg"
Sep 16 17:37:34.013768 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:34.013180    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txsj5\" (UniqueName: \"kubernetes.io/projected/356b448f-a555-4ede-a47c-c4be3e8444c8-kube-api-access-txsj5\") pod \"seaweedfs-64568bcd49-6wwcz\" (UID: \"356b448f-a555-4ede-a47c-c4be3e8444c8\") " pod="kserve/seaweedfs-64568bcd49-6wwcz"
Sep 16 17:37:34.057217 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:34.057147    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:37:34.068705 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:34.068681    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-6wwcz"
Sep 16 17:37:34.236973 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:34.236939    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:37:34.278798 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:34.278772    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-cc8594966-qs2df"]
Sep 16 17:37:34.279999 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:37:34.279971    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b24012_a56c_45d8_ad67_a1c9d9d282e6.slice/crio-5ec747710b23c98b781824d29b051dad7622bfc91e8b58c8fe0a5607eab04ca2 WatchSource:0}: Error finding container 5ec747710b23c98b781824d29b051dad7622bfc91e8b58c8fe0a5607eab04ca2: Status 404 returned error can't find the container with id 5ec747710b23c98b781824d29b051dad7622bfc91e8b58c8fe0a5607eab04ca2
Sep 16 17:37:34.291267 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:37:34.291245    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8a0f5af5_641b_43af_823b_e001c9ec9e88.slice/crio-bb4ba4f2257baff0a88d8fcf342e39e51c58f316deaf3a48edf1be09241c7d2d WatchSource:0}: Error finding container bb4ba4f2257baff0a88d8fcf342e39e51c58f316deaf3a48edf1be09241c7d2d: Status 404 returned error can't find the container with id bb4ba4f2257baff0a88d8fcf342e39e51c58f316deaf3a48edf1be09241c7d2d
Sep 16 17:37:34.293329 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:34.293302    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-5f574bd55d-xlhdg"]
Sep 16 17:37:34.344905 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:34.344877    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-6wwcz"]
Sep 16 17:37:34.346041 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:37:34.346019    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod356b448f_a555_4ede_a47c_c4be3e8444c8.slice/crio-d6464690e681f62a422a9d5e591b0a55f29e9b81b430ff48b19833dbf05543e6 WatchSource:0}: Error finding container d6464690e681f62a422a9d5e591b0a55f29e9b81b430ff48b19833dbf05543e6: Status 404 returned error can't find the container with id d6464690e681f62a422a9d5e591b0a55f29e9b81b430ff48b19833dbf05543e6
Sep 16 17:37:34.351855 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:34.351829    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-5f574bd55d-xlhdg" event={"ID":"8a0f5af5-641b-43af-823b-e001c9ec9e88","Type":"ContainerStarted","Data":"bb4ba4f2257baff0a88d8fcf342e39e51c58f316deaf3a48edf1be09241c7d2d"}
Sep 16 17:37:34.353034 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:34.353008    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cc8594966-qs2df" event={"ID":"17b24012-a56c-45d8-ad67-a1c9d9d282e6","Type":"ContainerStarted","Data":"5ec747710b23c98b781824d29b051dad7622bfc91e8b58c8fe0a5607eab04ca2"}
Sep 16 17:37:34.683704 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:34.683619    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:37:34.795067 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:34.795025    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:37:34.887027 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:34.886991    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:37:35.363265 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:35.363170    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-6wwcz" event={"ID":"356b448f-a555-4ede-a47c-c4be3e8444c8","Type":"ContainerStarted","Data":"d6464690e681f62a422a9d5e591b0a55f29e9b81b430ff48b19833dbf05543e6"}
Sep 16 17:37:38.981505 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:38.981479    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 17:37:39.394113 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:39.394072    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-6wwcz" event={"ID":"356b448f-a555-4ede-a47c-c4be3e8444c8","Type":"ContainerStarted","Data":"e9d39e4d677535a489c4d3c682cffa5ddad0721a28c92084e597203cf0b3ce8d"}
Sep 16 17:37:39.394318 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:39.394164    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-6wwcz"
Sep 16 17:37:39.396359 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:39.396293    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cc8594966-qs2df" event={"ID":"17b24012-a56c-45d8-ad67-a1c9d9d282e6","Type":"ContainerStarted","Data":"d091b2dac94d7358b8a09ebb27824d83ea872423f75d2f3be7cade644c54b4de"}
Sep 16 17:37:39.396498 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:39.396377    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-cc8594966-qs2df"
Sep 16 17:37:39.398170 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:39.398149    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-5f574bd55d-xlhdg" event={"ID":"8a0f5af5-641b-43af-823b-e001c9ec9e88","Type":"ContainerStarted","Data":"9bf2730f5d88b2a5184c56bae17497941768bd22cf83e2ae73ea9b1b06e5a3ec"}
Sep 16 17:37:39.398286 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:39.398262    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-5f574bd55d-xlhdg"
Sep 16 17:37:39.465108 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:39.465052    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-5f574bd55d-xlhdg" podStartSLOduration=1.8718088750000001 podStartE2EDuration="6.465036324s" totalImagesPullingTime="4.593227449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:37:39.463061723 +0000 UTC m=+780.825740535" watchObservedRunningTime="2026-09-16 17:37:39.465036324 +0000 UTC m=+780.827715109"
Sep 16 17:37:39.465259 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:39.465136    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-6wwcz" podStartSLOduration=1.834409003 podStartE2EDuration="6.465129682s" totalImagesPullingTime="4.630720679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:37:39.434019062 +0000 UTC m=+780.796697847" watchObservedRunningTime="2026-09-16 17:37:39.465129682 +0000 UTC m=+780.827808491"
Sep 16 17:37:39.512391 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:39.512331    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-cc8594966-qs2df" podStartSLOduration=2.018068662 podStartE2EDuration="6.512314302s" totalImagesPullingTime="4.49424564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:37:39.508625213 +0000 UTC m=+780.871304001" watchObservedRunningTime="2026-09-16 17:37:39.512314302 +0000 UTC m=+780.874993087"
Sep 16 17:37:45.406311 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:45.406276    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-6wwcz"
Sep 16 17:37:55.161663 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:37:55.161623    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:38:10.330830 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:10.330728    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:38:10.405938 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:10.405907    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-5f574bd55d-xlhdg"
Sep 16 17:38:10.408723 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:10.408701    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-cc8594966-qs2df"
Sep 16 17:38:20.651615 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:20.651562    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-cc8594966-qs2df"]
Sep 16 17:38:20.652041 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:20.651830    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-cc8594966-qs2df" podUID="17b24012-a56c-45d8-ad67-a1c9d9d282e6" containerName="manager" containerID="cri-o://d091b2dac94d7358b8a09ebb27824d83ea872423f75d2f3be7cade644c54b4de" gracePeriod=10
Sep 16 17:38:20.682810 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:20.682779    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-cc8594966-hqrxv"]
Sep 16 17:38:20.685170 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:20.685153    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cc8594966-hqrxv"
Sep 16 17:38:20.697721 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:20.697693    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-cc8594966-hqrxv"]
Sep 16 17:38:20.780847 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:20.780813    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpbkc\" (UniqueName: \"kubernetes.io/projected/85d8ff1b-c87d-4edf-8061-756fb2301955-kube-api-access-wpbkc\") pod \"kserve-controller-manager-cc8594966-hqrxv\" (UID: \"85d8ff1b-c87d-4edf-8061-756fb2301955\") " pod="kserve/kserve-controller-manager-cc8594966-hqrxv"
Sep 16 17:38:20.781037 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:20.780950    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85d8ff1b-c87d-4edf-8061-756fb2301955-cert\") pod \"kserve-controller-manager-cc8594966-hqrxv\" (UID: \"85d8ff1b-c87d-4edf-8061-756fb2301955\") " pod="kserve/kserve-controller-manager-cc8594966-hqrxv"
Sep 16 17:38:20.881417 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:20.881390    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpbkc\" (UniqueName: \"kubernetes.io/projected/85d8ff1b-c87d-4edf-8061-756fb2301955-kube-api-access-wpbkc\") pod \"kserve-controller-manager-cc8594966-hqrxv\" (UID: \"85d8ff1b-c87d-4edf-8061-756fb2301955\") " pod="kserve/kserve-controller-manager-cc8594966-hqrxv"
Sep 16 17:38:20.881522 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:20.881442    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85d8ff1b-c87d-4edf-8061-756fb2301955-cert\") pod \"kserve-controller-manager-cc8594966-hqrxv\" (UID: \"85d8ff1b-c87d-4edf-8061-756fb2301955\") " pod="kserve/kserve-controller-manager-cc8594966-hqrxv"
Sep 16 17:38:20.883850 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:20.883828    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85d8ff1b-c87d-4edf-8061-756fb2301955-cert\") pod \"kserve-controller-manager-cc8594966-hqrxv\" (UID: \"85d8ff1b-c87d-4edf-8061-756fb2301955\") " pod="kserve/kserve-controller-manager-cc8594966-hqrxv"
Sep 16 17:38:20.889909 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:20.889857    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpbkc\" (UniqueName: \"kubernetes.io/projected/85d8ff1b-c87d-4edf-8061-756fb2301955-kube-api-access-wpbkc\") pod \"kserve-controller-manager-cc8594966-hqrxv\" (UID: \"85d8ff1b-c87d-4edf-8061-756fb2301955\") " pod="kserve/kserve-controller-manager-cc8594966-hqrxv"
Sep 16 17:38:20.902975 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:20.902929    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cc8594966-qs2df"
Sep 16 17:38:20.982519 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:20.982484    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhsnb\" (UniqueName: \"kubernetes.io/projected/17b24012-a56c-45d8-ad67-a1c9d9d282e6-kube-api-access-dhsnb\") pod \"17b24012-a56c-45d8-ad67-a1c9d9d282e6\" (UID: \"17b24012-a56c-45d8-ad67-a1c9d9d282e6\") "
Sep 16 17:38:20.982709 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:20.982579    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17b24012-a56c-45d8-ad67-a1c9d9d282e6-cert\") pod \"17b24012-a56c-45d8-ad67-a1c9d9d282e6\" (UID: \"17b24012-a56c-45d8-ad67-a1c9d9d282e6\") "
Sep 16 17:38:20.984657 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:20.984622    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17b24012-a56c-45d8-ad67-a1c9d9d282e6-cert" (OuterVolumeSpecName: "cert") pod "17b24012-a56c-45d8-ad67-a1c9d9d282e6" (UID: "17b24012-a56c-45d8-ad67-a1c9d9d282e6"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:38:20.984771 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:20.984696    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b24012-a56c-45d8-ad67-a1c9d9d282e6-kube-api-access-dhsnb" (OuterVolumeSpecName: "kube-api-access-dhsnb") pod "17b24012-a56c-45d8-ad67-a1c9d9d282e6" (UID: "17b24012-a56c-45d8-ad67-a1c9d9d282e6"). InnerVolumeSpecName "kube-api-access-dhsnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:38:21.019904 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:21.019858    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cc8594966-hqrxv"
Sep 16 17:38:21.083186 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:21.083159    2577 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17b24012-a56c-45d8-ad67-a1c9d9d282e6-cert\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:38:21.083186 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:21.083188    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhsnb\" (UniqueName: \"kubernetes.io/projected/17b24012-a56c-45d8-ad67-a1c9d9d282e6-kube-api-access-dhsnb\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:38:21.292003 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:21.291976    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-cc8594966-hqrxv"]
Sep 16 17:38:21.293081 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:38:21.293052    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d8ff1b_c87d_4edf_8061_756fb2301955.slice/crio-f770ec751789c9935873cedc2fc1127966d64df8e8e1224cba3f90662c8afced WatchSource:0}: Error finding container f770ec751789c9935873cedc2fc1127966d64df8e8e1224cba3f90662c8afced: Status 404 returned error can't find the container with id f770ec751789c9935873cedc2fc1127966d64df8e8e1224cba3f90662c8afced
Sep 16 17:38:21.294517 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:21.294499    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 17:38:21.621616 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:21.621589    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cc8594966-hqrxv" event={"ID":"85d8ff1b-c87d-4edf-8061-756fb2301955","Type":"ContainerStarted","Data":"f770ec751789c9935873cedc2fc1127966d64df8e8e1224cba3f90662c8afced"}
Sep 16 17:38:21.622959 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:21.622934    2577 generic.go:356] "Generic (PLEG): container finished" podID="17b24012-a56c-45d8-ad67-a1c9d9d282e6" containerID="d091b2dac94d7358b8a09ebb27824d83ea872423f75d2f3be7cade644c54b4de" exitCode=0
Sep 16 17:38:21.623075 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:21.622994    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cc8594966-qs2df"
Sep 16 17:38:21.623075 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:21.623001    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cc8594966-qs2df" event={"ID":"17b24012-a56c-45d8-ad67-a1c9d9d282e6","Type":"ContainerDied","Data":"d091b2dac94d7358b8a09ebb27824d83ea872423f75d2f3be7cade644c54b4de"}
Sep 16 17:38:21.623075 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:21.623051    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cc8594966-qs2df" event={"ID":"17b24012-a56c-45d8-ad67-a1c9d9d282e6","Type":"ContainerDied","Data":"5ec747710b23c98b781824d29b051dad7622bfc91e8b58c8fe0a5607eab04ca2"}
Sep 16 17:38:21.623235 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:21.623079    2577 scope.go:117] "RemoveContainer" containerID="d091b2dac94d7358b8a09ebb27824d83ea872423f75d2f3be7cade644c54b4de"
Sep 16 17:38:21.638489 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:21.638469    2577 scope.go:117] "RemoveContainer" containerID="d091b2dac94d7358b8a09ebb27824d83ea872423f75d2f3be7cade644c54b4de"
Sep 16 17:38:21.638833 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:38:21.638813    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d091b2dac94d7358b8a09ebb27824d83ea872423f75d2f3be7cade644c54b4de\": container with ID starting with d091b2dac94d7358b8a09ebb27824d83ea872423f75d2f3be7cade644c54b4de not found: ID does not exist" containerID="d091b2dac94d7358b8a09ebb27824d83ea872423f75d2f3be7cade644c54b4de"
Sep 16 17:38:21.638958 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:21.638842    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d091b2dac94d7358b8a09ebb27824d83ea872423f75d2f3be7cade644c54b4de"} err="failed to get container status \"d091b2dac94d7358b8a09ebb27824d83ea872423f75d2f3be7cade644c54b4de\": rpc error: code = NotFound desc = could not find container \"d091b2dac94d7358b8a09ebb27824d83ea872423f75d2f3be7cade644c54b4de\": container with ID starting with d091b2dac94d7358b8a09ebb27824d83ea872423f75d2f3be7cade644c54b4de not found: ID does not exist"
Sep 16 17:38:21.651917 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:21.651894    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-cc8594966-qs2df"]
Sep 16 17:38:21.656126 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:21.656103    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-cc8594966-qs2df"]
Sep 16 17:38:22.631323 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:22.631285    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cc8594966-hqrxv" event={"ID":"85d8ff1b-c87d-4edf-8061-756fb2301955","Type":"ContainerStarted","Data":"b044703c05530365203e9aefd7fa05ca0b2e8a042fdf83564bd1223cc8ce5bf8"}
Sep 16 17:38:22.631509 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:22.631369    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-cc8594966-hqrxv"
Sep 16 17:38:22.654283 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:22.654227    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-cc8594966-hqrxv" podStartSLOduration=2.355962204 podStartE2EDuration="2.654211788s" totalImagesPullingTime="298.249584ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:38:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:38:22.653793165 +0000 UTC m=+824.016471950" watchObservedRunningTime="2026-09-16 17:38:22.654211788 +0000 UTC m=+824.016890574"
Sep 16 17:38:23.081045 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:23.081013    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17b24012-a56c-45d8-ad67-a1c9d9d282e6" path="/var/lib/kubelet/pods/17b24012-a56c-45d8-ad67-a1c9d9d282e6/volumes"
Sep 16 17:38:24.313745 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:24.313697    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:38:24.415693 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:24.415657    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:38:35.666914 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:35.666826    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:38:41.155387 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:41.155348    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:38:53.640556 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:53.640525    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-cc8594966-hqrxv"
Sep 16 17:38:55.112048 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.111991    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:38:55.162413 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.162374    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-2p94d"]
Sep 16 17:38:55.162751 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.162737    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17b24012-a56c-45d8-ad67-a1c9d9d282e6" containerName="manager"
Sep 16 17:38:55.162751 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.162751    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="17b24012-a56c-45d8-ad67-a1c9d9d282e6" containerName="manager"
Sep 16 17:38:55.162828 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.162807    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="17b24012-a56c-45d8-ad67-a1c9d9d282e6" containerName="manager"
Sep 16 17:38:55.166050 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.166030    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-2p94d"
Sep 16 17:38:55.176267 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.176247    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64f6302e-e847-45c3-ac32-db2947e8d9af-tls-certs\") pod \"model-serving-api-7b954ffc95-2p94d\" (UID: \"64f6302e-e847-45c3-ac32-db2947e8d9af\") " pod="kserve/model-serving-api-7b954ffc95-2p94d"
Sep 16 17:38:55.176353 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.176288    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kmvq\" (UniqueName: \"kubernetes.io/projected/64f6302e-e847-45c3-ac32-db2947e8d9af-kube-api-access-8kmvq\") pod \"model-serving-api-7b954ffc95-2p94d\" (UID: \"64f6302e-e847-45c3-ac32-db2947e8d9af\") " pod="kserve/model-serving-api-7b954ffc95-2p94d"
Sep 16 17:38:55.180813 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.180788    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-v2wdc\""
Sep 16 17:38:55.180813 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.180802    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 16 17:38:55.191673 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.191648    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-2p94d"]
Sep 16 17:38:55.194678 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.194656    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-hb6sq"]
Sep 16 17:38:55.197968 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.197953    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-hb6sq"
Sep 16 17:38:55.202140 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.202123    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 16 17:38:55.202244 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.202144    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 16 17:38:55.202244 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.202206    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-lqqh7\""
Sep 16 17:38:55.209127 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.209105    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-hb6sq"]
Sep 16 17:38:55.229757 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.229727    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:38:55.276939 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.276894    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54bdcc45-59ee-4c6e-857f-b2fa59ca861d-metrics-certs\") pod \"odh-model-controller-7b966d7984-hb6sq\" (UID: \"54bdcc45-59ee-4c6e-857f-b2fa59ca861d\") " pod="kserve/odh-model-controller-7b966d7984-hb6sq"
Sep 16 17:38:55.277103 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.276970    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5tjb\" (UniqueName: \"kubernetes.io/projected/54bdcc45-59ee-4c6e-857f-b2fa59ca861d-kube-api-access-t5tjb\") pod \"odh-model-controller-7b966d7984-hb6sq\" (UID: \"54bdcc45-59ee-4c6e-857f-b2fa59ca861d\") " pod="kserve/odh-model-controller-7b966d7984-hb6sq"
Sep 16 17:38:55.277103 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.277009    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64f6302e-e847-45c3-ac32-db2947e8d9af-tls-certs\") pod \"model-serving-api-7b954ffc95-2p94d\" (UID: \"64f6302e-e847-45c3-ac32-db2947e8d9af\") " pod="kserve/model-serving-api-7b954ffc95-2p94d"
Sep 16 17:38:55.277103 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.277027    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kmvq\" (UniqueName: \"kubernetes.io/projected/64f6302e-e847-45c3-ac32-db2947e8d9af-kube-api-access-8kmvq\") pod \"model-serving-api-7b954ffc95-2p94d\" (UID: \"64f6302e-e847-45c3-ac32-db2947e8d9af\") " pod="kserve/model-serving-api-7b954ffc95-2p94d"
Sep 16 17:38:55.277226 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.277120    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54bdcc45-59ee-4c6e-857f-b2fa59ca861d-cert\") pod \"odh-model-controller-7b966d7984-hb6sq\" (UID: \"54bdcc45-59ee-4c6e-857f-b2fa59ca861d\") " pod="kserve/odh-model-controller-7b966d7984-hb6sq"
Sep 16 17:38:55.277226 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:38:55.277177    2577 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 16 17:38:55.277300 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:38:55.277243    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64f6302e-e847-45c3-ac32-db2947e8d9af-tls-certs podName:64f6302e-e847-45c3-ac32-db2947e8d9af nodeName:}" failed. No retries permitted until 2026-09-16 17:38:55.777223662 +0000 UTC m=+857.139902436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/64f6302e-e847-45c3-ac32-db2947e8d9af-tls-certs") pod "model-serving-api-7b954ffc95-2p94d" (UID: "64f6302e-e847-45c3-ac32-db2947e8d9af") : secret "model-serving-api-tls" not found
Sep 16 17:38:55.289220 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.289197    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kmvq\" (UniqueName: \"kubernetes.io/projected/64f6302e-e847-45c3-ac32-db2947e8d9af-kube-api-access-8kmvq\") pod \"model-serving-api-7b954ffc95-2p94d\" (UID: \"64f6302e-e847-45c3-ac32-db2947e8d9af\") " pod="kserve/model-serving-api-7b954ffc95-2p94d"
Sep 16 17:38:55.322130 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.322102    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:38:55.378369 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.378285    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54bdcc45-59ee-4c6e-857f-b2fa59ca861d-cert\") pod \"odh-model-controller-7b966d7984-hb6sq\" (UID: \"54bdcc45-59ee-4c6e-857f-b2fa59ca861d\") " pod="kserve/odh-model-controller-7b966d7984-hb6sq"
Sep 16 17:38:55.378369 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.378330    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54bdcc45-59ee-4c6e-857f-b2fa59ca861d-metrics-certs\") pod \"odh-model-controller-7b966d7984-hb6sq\" (UID: \"54bdcc45-59ee-4c6e-857f-b2fa59ca861d\") " pod="kserve/odh-model-controller-7b966d7984-hb6sq"
Sep 16 17:38:55.378369 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.378363    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5tjb\" (UniqueName: \"kubernetes.io/projected/54bdcc45-59ee-4c6e-857f-b2fa59ca861d-kube-api-access-t5tjb\") pod \"odh-model-controller-7b966d7984-hb6sq\" (UID: \"54bdcc45-59ee-4c6e-857f-b2fa59ca861d\") " pod="kserve/odh-model-controller-7b966d7984-hb6sq"
Sep 16 17:38:55.380981 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.380955    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54bdcc45-59ee-4c6e-857f-b2fa59ca861d-metrics-certs\") pod \"odh-model-controller-7b966d7984-hb6sq\" (UID: \"54bdcc45-59ee-4c6e-857f-b2fa59ca861d\") " pod="kserve/odh-model-controller-7b966d7984-hb6sq"
Sep 16 17:38:55.381098 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.380955    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54bdcc45-59ee-4c6e-857f-b2fa59ca861d-cert\") pod \"odh-model-controller-7b966d7984-hb6sq\" (UID: \"54bdcc45-59ee-4c6e-857f-b2fa59ca861d\") " pod="kserve/odh-model-controller-7b966d7984-hb6sq"
Sep 16 17:38:55.388071 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.388048    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5tjb\" (UniqueName: \"kubernetes.io/projected/54bdcc45-59ee-4c6e-857f-b2fa59ca861d-kube-api-access-t5tjb\") pod \"odh-model-controller-7b966d7984-hb6sq\" (UID: \"54bdcc45-59ee-4c6e-857f-b2fa59ca861d\") " pod="kserve/odh-model-controller-7b966d7984-hb6sq"
Sep 16 17:38:55.433972 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.433938    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:38:55.508079 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.508047    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-hb6sq"
Sep 16 17:38:55.638535 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.638434    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:38:55.781511 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.781480    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64f6302e-e847-45c3-ac32-db2947e8d9af-tls-certs\") pod \"model-serving-api-7b954ffc95-2p94d\" (UID: \"64f6302e-e847-45c3-ac32-db2947e8d9af\") " pod="kserve/model-serving-api-7b954ffc95-2p94d"
Sep 16 17:38:55.785013 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.783771    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-hb6sq"]
Sep 16 17:38:55.785013 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.784197    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64f6302e-e847-45c3-ac32-db2947e8d9af-tls-certs\") pod \"model-serving-api-7b954ffc95-2p94d\" (UID: \"64f6302e-e847-45c3-ac32-db2947e8d9af\") " pod="kserve/model-serving-api-7b954ffc95-2p94d"
Sep 16 17:38:55.805814 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:55.805788    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-hb6sq" event={"ID":"54bdcc45-59ee-4c6e-857f-b2fa59ca861d","Type":"ContainerStarted","Data":"834c66008d88266befb4223074b6e5977a8f26d6414e9adcc096018f5f1dfdbd"}
Sep 16 17:38:56.075305 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:56.075279    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-2p94d"
Sep 16 17:38:56.354108 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:38:56.354068    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f6302e_e847_45c3_ac32_db2947e8d9af.slice/crio-53096fe13d7008188f4ebb56cb4c86dc453a33831b5574f86e1a13f462229fdb WatchSource:0}: Error finding container 53096fe13d7008188f4ebb56cb4c86dc453a33831b5574f86e1a13f462229fdb: Status 404 returned error can't find the container with id 53096fe13d7008188f4ebb56cb4c86dc453a33831b5574f86e1a13f462229fdb
Sep 16 17:38:56.354888 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:56.354782    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-2p94d"]
Sep 16 17:38:56.607554 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:56.607466    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:38:56.689953 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:56.689900    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:38:56.812266 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:56.812229    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-2p94d" event={"ID":"64f6302e-e847-45c3-ac32-db2947e8d9af","Type":"ContainerStarted","Data":"53096fe13d7008188f4ebb56cb4c86dc453a33831b5574f86e1a13f462229fdb"}
Sep 16 17:38:59.834107 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:59.834055    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-hb6sq" event={"ID":"54bdcc45-59ee-4c6e-857f-b2fa59ca861d","Type":"ContainerStarted","Data":"619c16a2422b5b16fea66a22f7be1e8d019bfb17b1a1858b52472f9659c8b857"}
Sep 16 17:38:59.834640 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:59.834145    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-hb6sq"
Sep 16 17:38:59.835949 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:59.835924    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-2p94d" event={"ID":"64f6302e-e847-45c3-ac32-db2947e8d9af","Type":"ContainerStarted","Data":"cbd71239cc3c9b9bd0e9cea4d166656c353f931c23a500cf1b752100a6d54820"}
Sep 16 17:38:59.836090 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:59.836012    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-2p94d"
Sep 16 17:38:59.868153 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:59.868081    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-hb6sq" podStartSLOduration=1.139696871 podStartE2EDuration="4.868066851s" totalImagesPullingTime="3.72836998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:38:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:38:59.866088893 +0000 UTC m=+861.228767679" watchObservedRunningTime="2026-09-16 17:38:59.868066851 +0000 UTC m=+861.230745683"
Sep 16 17:38:59.897433 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:38:59.897368    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-2p94d" podStartSLOduration=1.699517765 podStartE2EDuration="4.897345775s" totalImagesPullingTime="3.19782801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:38:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:38:59.896137743 +0000 UTC m=+861.258816530" watchObservedRunningTime="2026-09-16 17:38:59.897345775 +0000 UTC m=+861.260024562"
Sep 16 17:39:10.843986 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:39:10.843952    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-hb6sq"
Sep 16 17:39:10.845567 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:39:10.845549    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-2p94d"
Sep 16 17:39:11.678443 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:39:11.678404    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-ppnh5"]
Sep 16 17:39:11.684739 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:39:11.684719    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-ppnh5"
Sep 16 17:39:11.687364 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:39:11.687342    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 17:39:11.694906 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:39:11.694882    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-ppnh5"]
Sep 16 17:39:11.716567 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:39:11.716524    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfvzm\" (UniqueName: \"kubernetes.io/projected/7236084f-2910-4707-954e-02331587c5e2-kube-api-access-dfvzm\") pod \"s3-init-ppnh5\" (UID: \"7236084f-2910-4707-954e-02331587c5e2\") " pod="kserve/s3-init-ppnh5"
Sep 16 17:39:11.716702 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:39:11.716577    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7236084f-2910-4707-954e-02331587c5e2-model-cache\") pod \"s3-init-ppnh5\" (UID: \"7236084f-2910-4707-954e-02331587c5e2\") " pod="kserve/s3-init-ppnh5"
Sep 16 17:39:11.817928 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:39:11.817890    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfvzm\" (UniqueName: \"kubernetes.io/projected/7236084f-2910-4707-954e-02331587c5e2-kube-api-access-dfvzm\") pod \"s3-init-ppnh5\" (UID: \"7236084f-2910-4707-954e-02331587c5e2\") " pod="kserve/s3-init-ppnh5"
Sep 16 17:39:11.818101 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:39:11.817946    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7236084f-2910-4707-954e-02331587c5e2-model-cache\") pod \"s3-init-ppnh5\" (UID: \"7236084f-2910-4707-954e-02331587c5e2\") " pod="kserve/s3-init-ppnh5"
Sep 16 17:39:11.818310 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:39:11.818295    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7236084f-2910-4707-954e-02331587c5e2-model-cache\") pod \"s3-init-ppnh5\" (UID: \"7236084f-2910-4707-954e-02331587c5e2\") " pod="kserve/s3-init-ppnh5"
Sep 16 17:39:11.827523 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:39:11.827491    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfvzm\" (UniqueName: \"kubernetes.io/projected/7236084f-2910-4707-954e-02331587c5e2-kube-api-access-dfvzm\") pod \"s3-init-ppnh5\" (UID: \"7236084f-2910-4707-954e-02331587c5e2\") " pod="kserve/s3-init-ppnh5"
Sep 16 17:39:11.994509 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:39:11.994427    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-ppnh5"
Sep 16 17:39:12.276817 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:39:12.276788    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-ppnh5"]
Sep 16 17:39:12.279743 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:39:12.279711    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7236084f_2910_4707_954e_02331587c5e2.slice/crio-02514f2bf00a80b4bce51ec0228a4a1a375e47d3a5a2185768fd709523f512fb WatchSource:0}: Error finding container 02514f2bf00a80b4bce51ec0228a4a1a375e47d3a5a2185768fd709523f512fb: Status 404 returned error can't find the container with id 02514f2bf00a80b4bce51ec0228a4a1a375e47d3a5a2185768fd709523f512fb
Sep 16 17:39:12.905616 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:39:12.905567    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ppnh5" event={"ID":"7236084f-2910-4707-954e-02331587c5e2","Type":"ContainerStarted","Data":"02514f2bf00a80b4bce51ec0228a4a1a375e47d3a5a2185768fd709523f512fb"}
Sep 16 17:39:15.925132 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:39:15.925044    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ppnh5" event={"ID":"7236084f-2910-4707-954e-02331587c5e2","Type":"ContainerStarted","Data":"3b161534db553d2420ebb85a973ab791d2f01d2ce286948abb1a6537d89c4c53"}
Sep 16 17:39:17.162145 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:39:17.162114    2577 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7236084f_2910_4707_954e_02331587c5e2.slice/crio-3b161534db553d2420ebb85a973ab791d2f01d2ce286948abb1a6537d89c4c53.scope\": RecentStats: unable to find data in memory cache]"
Sep 16 17:39:17.938416 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:39:17.938384    2577 generic.go:356] "Generic (PLEG): container finished" podID="7236084f-2910-4707-954e-02331587c5e2" containerID="3b161534db553d2420ebb85a973ab791d2f01d2ce286948abb1a6537d89c4c53" exitCode=0
Sep 16 17:39:17.938585 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:39:17.938462    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ppnh5" event={"ID":"7236084f-2910-4707-954e-02331587c5e2","Type":"ContainerDied","Data":"3b161534db553d2420ebb85a973ab791d2f01d2ce286948abb1a6537d89c4c53"}
Sep 16 17:39:22.971366 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:39:22.971328    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ppnh5" event={"ID":"7236084f-2910-4707-954e-02331587c5e2","Type":"ContainerStarted","Data":"2dcaca7a489a93f917ed8868eb586bbb30664da6730041784e8192f612d3117f"}
Sep 16 17:39:22.993197 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:39:22.993144    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-ppnh5" podStartSLOduration=2.645855785 podStartE2EDuration="11.993124216s" totalImagesPullingTime="7.841505682s" totalInitContainerRuntime="1.505762749s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:39:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:39:22.992475147 +0000 UTC m=+884.355153934" watchObservedRunningTime="2026-09-16 17:39:22.993124216 +0000 UTC m=+884.355803003"
Sep 16 17:40:28.307110 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:28.307078    2577 generic.go:356] "Generic (PLEG): container finished" podID="7236084f-2910-4707-954e-02331587c5e2" containerID="2dcaca7a489a93f917ed8868eb586bbb30664da6730041784e8192f612d3117f" exitCode=0
Sep 16 17:40:28.307502 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:28.307158    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ppnh5" event={"ID":"7236084f-2910-4707-954e-02331587c5e2","Type":"ContainerDied","Data":"2dcaca7a489a93f917ed8868eb586bbb30664da6730041784e8192f612d3117f"}
Sep 16 17:40:29.447341 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:29.447314    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-ppnh5"
Sep 16 17:40:29.586842 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:29.586760    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfvzm\" (UniqueName: \"kubernetes.io/projected/7236084f-2910-4707-954e-02331587c5e2-kube-api-access-dfvzm\") pod \"7236084f-2910-4707-954e-02331587c5e2\" (UID: \"7236084f-2910-4707-954e-02331587c5e2\") "
Sep 16 17:40:29.586842 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:29.586814    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7236084f-2910-4707-954e-02331587c5e2-model-cache\") pod \"7236084f-2910-4707-954e-02331587c5e2\" (UID: \"7236084f-2910-4707-954e-02331587c5e2\") "
Sep 16 17:40:29.588880 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:29.588841    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7236084f-2910-4707-954e-02331587c5e2-kube-api-access-dfvzm" (OuterVolumeSpecName: "kube-api-access-dfvzm") pod "7236084f-2910-4707-954e-02331587c5e2" (UID: "7236084f-2910-4707-954e-02331587c5e2"). InnerVolumeSpecName "kube-api-access-dfvzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:40:29.609371 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:29.609325    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7236084f-2910-4707-954e-02331587c5e2-model-cache" (OuterVolumeSpecName: "model-cache") pod "7236084f-2910-4707-954e-02331587c5e2" (UID: "7236084f-2910-4707-954e-02331587c5e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:40:29.687599 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:29.687559    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfvzm\" (UniqueName: \"kubernetes.io/projected/7236084f-2910-4707-954e-02331587c5e2-kube-api-access-dfvzm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:40:29.687599 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:29.687594    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7236084f-2910-4707-954e-02331587c5e2-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:40:30.318569 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:30.318539    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-ppnh5"
Sep 16 17:40:30.318746 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:30.318539    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ppnh5" event={"ID":"7236084f-2910-4707-954e-02331587c5e2","Type":"ContainerDied","Data":"02514f2bf00a80b4bce51ec0228a4a1a375e47d3a5a2185768fd709523f512fb"}
Sep 16 17:40:30.318746 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:30.318667    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02514f2bf00a80b4bce51ec0228a4a1a375e47d3a5a2185768fd709523f512fb"
Sep 16 17:40:30.763641 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:30.763561    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:40:30.878445 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:30.878404    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:40:44.033405 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:44.033369    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:40:44.232347 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:44.232312    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:40:51.772599 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:51.772558    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct"]
Sep 16 17:40:51.773171 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:51.773150    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7236084f-2910-4707-954e-02331587c5e2" containerName="download-hf-model"
Sep 16 17:40:51.773215 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:51.773174    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7236084f-2910-4707-954e-02331587c5e2" containerName="download-hf-model"
Sep 16 17:40:51.773215 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:51.773188    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7236084f-2910-4707-954e-02331587c5e2" containerName="s3-init"
Sep 16 17:40:51.773215 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:51.773197    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7236084f-2910-4707-954e-02331587c5e2" containerName="s3-init"
Sep 16 17:40:51.773310 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:51.773300    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7236084f-2910-4707-954e-02331587c5e2" containerName="s3-init"
Sep 16 17:40:51.776612 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:51.776592    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct"
Sep 16 17:40:51.780815 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:51.780795    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-8lrqs\""
Sep 16 17:40:51.780815 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:51.780808    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 16 17:40:51.781011 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:51.780809    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 16 17:40:51.781011 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:51.780824    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 16 17:40:51.787049 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:51.787030    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct"]
Sep 16 17:40:51.799680 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:51.799639    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:40:51.874049 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:51.874013    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/102afe40-2887-4907-8ed8-c00b39339baa-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-g9pct\" (UID: \"102afe40-2887-4907-8ed8-c00b39339baa\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct"
Sep 16 17:40:51.874049 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:51.874051    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/102afe40-2887-4907-8ed8-c00b39339baa-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-g9pct\" (UID: \"102afe40-2887-4907-8ed8-c00b39339baa\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct"
Sep 16 17:40:51.874301 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:51.874078    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/102afe40-2887-4907-8ed8-c00b39339baa-dshm\") pod \"fc-smoke-test-kserve-7f4687897-g9pct\" (UID: \"102afe40-2887-4907-8ed8-c00b39339baa\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct"
Sep 16 17:40:51.874301 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:51.874106    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/102afe40-2887-4907-8ed8-c00b39339baa-home\") pod \"fc-smoke-test-kserve-7f4687897-g9pct\" (UID: \"102afe40-2887-4907-8ed8-c00b39339baa\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct"
Sep 16 17:40:51.874301 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:51.874125    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/102afe40-2887-4907-8ed8-c00b39339baa-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-g9pct\" (UID: \"102afe40-2887-4907-8ed8-c00b39339baa\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct"
Sep 16 17:40:51.874301 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:51.874242    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p88nh\" (UniqueName: \"kubernetes.io/projected/102afe40-2887-4907-8ed8-c00b39339baa-kube-api-access-p88nh\") pod \"fc-smoke-test-kserve-7f4687897-g9pct\" (UID: \"102afe40-2887-4907-8ed8-c00b39339baa\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct"
Sep 16 17:40:51.975376 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:51.975332    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p88nh\" (UniqueName: \"kubernetes.io/projected/102afe40-2887-4907-8ed8-c00b39339baa-kube-api-access-p88nh\") pod \"fc-smoke-test-kserve-7f4687897-g9pct\" (UID: \"102afe40-2887-4907-8ed8-c00b39339baa\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct"
Sep 16 17:40:51.975575 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:51.975393    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/102afe40-2887-4907-8ed8-c00b39339baa-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-g9pct\" (UID: \"102afe40-2887-4907-8ed8-c00b39339baa\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct"
Sep 16 17:40:51.975575 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:51.975418    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/102afe40-2887-4907-8ed8-c00b39339baa-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-g9pct\" (UID: \"102afe40-2887-4907-8ed8-c00b39339baa\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct"
Sep 16 17:40:51.975575 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:51.975442    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/102afe40-2887-4907-8ed8-c00b39339baa-dshm\") pod \"fc-smoke-test-kserve-7f4687897-g9pct\" (UID: \"102afe40-2887-4907-8ed8-c00b39339baa\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct"
Sep 16 17:40:51.975575 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:51.975475    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/102afe40-2887-4907-8ed8-c00b39339baa-home\") pod \"fc-smoke-test-kserve-7f4687897-g9pct\" (UID: \"102afe40-2887-4907-8ed8-c00b39339baa\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct"
Sep 16 17:40:51.975575 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:51.975499    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/102afe40-2887-4907-8ed8-c00b39339baa-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-g9pct\" (UID: \"102afe40-2887-4907-8ed8-c00b39339baa\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct"
Sep 16 17:40:51.975964 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:51.975942    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/102afe40-2887-4907-8ed8-c00b39339baa-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-g9pct\" (UID: \"102afe40-2887-4907-8ed8-c00b39339baa\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct"
Sep 16 17:40:51.976031 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:51.975958    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/102afe40-2887-4907-8ed8-c00b39339baa-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-g9pct\" (UID: \"102afe40-2887-4907-8ed8-c00b39339baa\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct"
Sep 16 17:40:51.976031 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:51.975988    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/102afe40-2887-4907-8ed8-c00b39339baa-home\") pod \"fc-smoke-test-kserve-7f4687897-g9pct\" (UID: \"102afe40-2887-4907-8ed8-c00b39339baa\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct"
Sep 16 17:40:51.977763 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:51.977740    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/102afe40-2887-4907-8ed8-c00b39339baa-dshm\") pod \"fc-smoke-test-kserve-7f4687897-g9pct\" (UID: \"102afe40-2887-4907-8ed8-c00b39339baa\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct"
Sep 16 17:40:51.977970 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:51.977951    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/102afe40-2887-4907-8ed8-c00b39339baa-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-g9pct\" (UID: \"102afe40-2887-4907-8ed8-c00b39339baa\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct"
Sep 16 17:40:51.984172 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:51.984153    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p88nh\" (UniqueName: \"kubernetes.io/projected/102afe40-2887-4907-8ed8-c00b39339baa-kube-api-access-p88nh\") pod \"fc-smoke-test-kserve-7f4687897-g9pct\" (UID: \"102afe40-2887-4907-8ed8-c00b39339baa\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct"
Sep 16 17:40:52.087384 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:52.087344    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct"
Sep 16 17:40:52.188995 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:52.188960    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg"]
Sep 16 17:40:52.193099 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:52.193074    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg"
Sep 16 17:40:52.195987 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:52.195924    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-b5rts\""
Sep 16 17:40:52.205174 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:52.205148    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg"]
Sep 16 17:40:52.218775 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:52.218752    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:40:52.277966 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:52.277925    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0547c48a-584d-4972-84f2-53b616ff1075-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg\" (UID: \"0547c48a-584d-4972-84f2-53b616ff1075\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg"
Sep 16 17:40:52.278181 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:52.277989    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl69b\" (UniqueName: \"kubernetes.io/projected/0547c48a-584d-4972-84f2-53b616ff1075-kube-api-access-fl69b\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg\" (UID: \"0547c48a-584d-4972-84f2-53b616ff1075\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg"
Sep 16 17:40:52.364531 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:52.364507    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct"]
Sep 16 17:40:52.366229 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:40:52.366204    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod102afe40_2887_4907_8ed8_c00b39339baa.slice/crio-25ee5362662ab52757398fd5b1352ca26637a8f98403f5c1ef09e84684042de5 WatchSource:0}: Error finding container 25ee5362662ab52757398fd5b1352ca26637a8f98403f5c1ef09e84684042de5: Status 404 returned error can't find the container with id 25ee5362662ab52757398fd5b1352ca26637a8f98403f5c1ef09e84684042de5
Sep 16 17:40:52.378711 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:52.378689    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl69b\" (UniqueName: \"kubernetes.io/projected/0547c48a-584d-4972-84f2-53b616ff1075-kube-api-access-fl69b\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg\" (UID: \"0547c48a-584d-4972-84f2-53b616ff1075\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg"
Sep 16 17:40:52.378807 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:52.378757    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0547c48a-584d-4972-84f2-53b616ff1075-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg\" (UID: \"0547c48a-584d-4972-84f2-53b616ff1075\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg"
Sep 16 17:40:52.381076 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:52.381059    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0547c48a-584d-4972-84f2-53b616ff1075-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg\" (UID: \"0547c48a-584d-4972-84f2-53b616ff1075\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg"
Sep 16 17:40:52.388065 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:52.388048    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl69b\" (UniqueName: \"kubernetes.io/projected/0547c48a-584d-4972-84f2-53b616ff1075-kube-api-access-fl69b\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg\" (UID: \"0547c48a-584d-4972-84f2-53b616ff1075\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg"
Sep 16 17:40:52.446241 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:52.446204    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct" event={"ID":"102afe40-2887-4907-8ed8-c00b39339baa","Type":"ContainerStarted","Data":"25ee5362662ab52757398fd5b1352ca26637a8f98403f5c1ef09e84684042de5"}
Sep 16 17:40:52.505754 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:52.505724    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg"
Sep 16 17:40:52.784951 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:52.784925    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg"]
Sep 16 17:40:52.786725 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:40:52.786697    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0547c48a_584d_4972_84f2_53b616ff1075.slice/crio-d03b511ac5585a670c01aba9eddda872c2f426c47b2de4d274269f23b8cd6a3c WatchSource:0}: Error finding container d03b511ac5585a670c01aba9eddda872c2f426c47b2de4d274269f23b8cd6a3c: Status 404 returned error can't find the container with id d03b511ac5585a670c01aba9eddda872c2f426c47b2de4d274269f23b8cd6a3c
Sep 16 17:40:53.454040 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:53.454005    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg" event={"ID":"0547c48a-584d-4972-84f2-53b616ff1075","Type":"ContainerStarted","Data":"d03b511ac5585a670c01aba9eddda872c2f426c47b2de4d274269f23b8cd6a3c"}
Sep 16 17:40:54.461427 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:54.461333    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct" event={"ID":"102afe40-2887-4907-8ed8-c00b39339baa","Type":"ContainerStarted","Data":"879b914ade7228721132e09b26831e5a2755acd82e05d51c5ac9c044f97571c2"}
Sep 16 17:40:54.501010 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:54.500770    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct" podStartSLOduration=1.722983085 podStartE2EDuration="3.50075143s" totalImagesPullingTime="1.777768345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:40:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:40:54.496881059 +0000 UTC m=+975.859559836" watchObservedRunningTime="2026-09-16 17:40:54.50075143 +0000 UTC m=+975.863430216"
Sep 16 17:40:55.088322 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:55.088286    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct"
Sep 16 17:40:56.477085 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:56.477052    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg" event={"ID":"0547c48a-584d-4972-84f2-53b616ff1075","Type":"ContainerStarted","Data":"c3d2f01d5a517290f904d21423e37c39c90eacf659f19bd049e927a04291a1aa"}
Sep 16 17:40:56.477469 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:56.477226    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg"
Sep 16 17:40:56.499206 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:40:56.499157    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg" podStartSLOduration=1.8711048940000001 podStartE2EDuration="4.49914096s" totalImagesPullingTime="2.628036066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:40:56.496937313 +0000 UTC m=+977.859616100" watchObservedRunningTime="2026-09-16 17:40:56.49914096 +0000 UTC m=+977.861819746"
Sep 16 17:41:02.088383 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:02.088284    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct"
Sep 16 17:41:02.101364 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:02.101339    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct"
Sep 16 17:41:02.521625 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:02.521535    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct"
Sep 16 17:41:02.606056 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:02.606022    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:41:02.703307 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:02.703270    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:41:03.686803 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:03.686762    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:41:20.088571 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:20.088512    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:41:27.484892 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:27.484845    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg"
Sep 16 17:41:34.688965 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:34.688932    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg"]
Sep 16 17:41:34.693240 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:34.693213    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct"]
Sep 16 17:41:34.702347 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:41:34.702305    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 17:41:34.702347 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:41:34.702347 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 17:41:34.702347 ip-10-0-136-122 kubenswrapper[2577]:  > containerID="879b914ade7228721132e09b26831e5a2755acd82e05d51c5ac9c044f97571c2" cmd=["/bin/sleep","15"]
Sep 16 17:41:34.702655 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:34.702378    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct" message="" err=<
Sep 16 17:41:34.702655 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:41:34.702655 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 17:41:34.702655 ip-10-0-136-122 kubenswrapper[2577]:  >
Sep 16 17:41:34.702655 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:41:34.702397    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 17:41:34.702655 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:41:34.702655 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 17:41:34.702655 ip-10-0-136-122 kubenswrapper[2577]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct" podUID="102afe40-2887-4907-8ed8-c00b39339baa" containerName="main" containerID="cri-o://879b914ade7228721132e09b26831e5a2755acd82e05d51c5ac9c044f97571c2"
Sep 16 17:41:34.702655 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:34.702450    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct" podUID="102afe40-2887-4907-8ed8-c00b39339baa" containerName="main" containerID="cri-o://879b914ade7228721132e09b26831e5a2755acd82e05d51c5ac9c044f97571c2" gracePeriod=60
Sep 16 17:41:34.707108 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:34.707084    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:41:34.776543 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:34.776510    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:41:35.061007 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:35.060985    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct"
Sep 16 17:41:35.163437 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:35.163402    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/102afe40-2887-4907-8ed8-c00b39339baa-dshm\") pod \"102afe40-2887-4907-8ed8-c00b39339baa\" (UID: \"102afe40-2887-4907-8ed8-c00b39339baa\") "
Sep 16 17:41:35.163612 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:35.163465    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/102afe40-2887-4907-8ed8-c00b39339baa-home\") pod \"102afe40-2887-4907-8ed8-c00b39339baa\" (UID: \"102afe40-2887-4907-8ed8-c00b39339baa\") "
Sep 16 17:41:35.163612 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:35.163492    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p88nh\" (UniqueName: \"kubernetes.io/projected/102afe40-2887-4907-8ed8-c00b39339baa-kube-api-access-p88nh\") pod \"102afe40-2887-4907-8ed8-c00b39339baa\" (UID: \"102afe40-2887-4907-8ed8-c00b39339baa\") "
Sep 16 17:41:35.163612 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:35.163516    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/102afe40-2887-4907-8ed8-c00b39339baa-model-cache\") pod \"102afe40-2887-4907-8ed8-c00b39339baa\" (UID: \"102afe40-2887-4907-8ed8-c00b39339baa\") "
Sep 16 17:41:35.163612 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:35.163552    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/102afe40-2887-4907-8ed8-c00b39339baa-tmp-dir\") pod \"102afe40-2887-4907-8ed8-c00b39339baa\" (UID: \"102afe40-2887-4907-8ed8-c00b39339baa\") "
Sep 16 17:41:35.163612 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:35.163593    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/102afe40-2887-4907-8ed8-c00b39339baa-tls-certs\") pod \"102afe40-2887-4907-8ed8-c00b39339baa\" (UID: \"102afe40-2887-4907-8ed8-c00b39339baa\") "
Sep 16 17:41:35.163916 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:35.163737    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/102afe40-2887-4907-8ed8-c00b39339baa-home" (OuterVolumeSpecName: "home") pod "102afe40-2887-4907-8ed8-c00b39339baa" (UID: "102afe40-2887-4907-8ed8-c00b39339baa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:41:35.163916 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:35.163838    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/102afe40-2887-4907-8ed8-c00b39339baa-model-cache" (OuterVolumeSpecName: "model-cache") pod "102afe40-2887-4907-8ed8-c00b39339baa" (UID: "102afe40-2887-4907-8ed8-c00b39339baa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:41:35.164011 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:35.163905    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/102afe40-2887-4907-8ed8-c00b39339baa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "102afe40-2887-4907-8ed8-c00b39339baa" (UID: "102afe40-2887-4907-8ed8-c00b39339baa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:41:35.164011 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:35.163983    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/102afe40-2887-4907-8ed8-c00b39339baa-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:41:35.164011 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:35.164004    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/102afe40-2887-4907-8ed8-c00b39339baa-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:41:35.165818 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:35.165789    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/102afe40-2887-4907-8ed8-c00b39339baa-dshm" (OuterVolumeSpecName: "dshm") pod "102afe40-2887-4907-8ed8-c00b39339baa" (UID: "102afe40-2887-4907-8ed8-c00b39339baa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:41:35.165969 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:35.165843    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/102afe40-2887-4907-8ed8-c00b39339baa-kube-api-access-p88nh" (OuterVolumeSpecName: "kube-api-access-p88nh") pod "102afe40-2887-4907-8ed8-c00b39339baa" (UID: "102afe40-2887-4907-8ed8-c00b39339baa"). InnerVolumeSpecName "kube-api-access-p88nh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:41:35.165969 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:35.165842    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/102afe40-2887-4907-8ed8-c00b39339baa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "102afe40-2887-4907-8ed8-c00b39339baa" (UID: "102afe40-2887-4907-8ed8-c00b39339baa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:41:35.264806 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:35.264721    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/102afe40-2887-4907-8ed8-c00b39339baa-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:41:35.264806 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:35.264749    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/102afe40-2887-4907-8ed8-c00b39339baa-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:41:35.264806 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:35.264757    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p88nh\" (UniqueName: \"kubernetes.io/projected/102afe40-2887-4907-8ed8-c00b39339baa-kube-api-access-p88nh\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:41:35.264806 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:35.264768    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/102afe40-2887-4907-8ed8-c00b39339baa-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:41:35.691489 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:35.691453    2577 generic.go:356] "Generic (PLEG): container finished" podID="102afe40-2887-4907-8ed8-c00b39339baa" containerID="879b914ade7228721132e09b26831e5a2755acd82e05d51c5ac9c044f97571c2" exitCode=0
Sep 16 17:41:35.691982 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:35.691501    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct" event={"ID":"102afe40-2887-4907-8ed8-c00b39339baa","Type":"ContainerDied","Data":"879b914ade7228721132e09b26831e5a2755acd82e05d51c5ac9c044f97571c2"}
Sep 16 17:41:35.691982 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:35.691519    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct"
Sep 16 17:41:35.691982 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:35.691553    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct" event={"ID":"102afe40-2887-4907-8ed8-c00b39339baa","Type":"ContainerDied","Data":"25ee5362662ab52757398fd5b1352ca26637a8f98403f5c1ef09e84684042de5"}
Sep 16 17:41:35.691982 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:35.691582    2577 scope.go:117] "RemoveContainer" containerID="879b914ade7228721132e09b26831e5a2755acd82e05d51c5ac9c044f97571c2"
Sep 16 17:41:35.700969 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:35.700950    2577 scope.go:117] "RemoveContainer" containerID="879b914ade7228721132e09b26831e5a2755acd82e05d51c5ac9c044f97571c2"
Sep 16 17:41:35.701248 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:41:35.701230    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"879b914ade7228721132e09b26831e5a2755acd82e05d51c5ac9c044f97571c2\": container with ID starting with 879b914ade7228721132e09b26831e5a2755acd82e05d51c5ac9c044f97571c2 not found: ID does not exist" containerID="879b914ade7228721132e09b26831e5a2755acd82e05d51c5ac9c044f97571c2"
Sep 16 17:41:35.701303 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:35.701256    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"879b914ade7228721132e09b26831e5a2755acd82e05d51c5ac9c044f97571c2"} err="failed to get container status \"879b914ade7228721132e09b26831e5a2755acd82e05d51c5ac9c044f97571c2\": rpc error: code = NotFound desc = could not find container \"879b914ade7228721132e09b26831e5a2755acd82e05d51c5ac9c044f97571c2\": container with ID starting with 879b914ade7228721132e09b26831e5a2755acd82e05d51c5ac9c044f97571c2 not found: ID does not exist"
Sep 16 17:41:35.725578 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:35.725551    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct"]
Sep 16 17:41:35.730174 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:35.730146    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-g9pct"]
Sep 16 17:41:37.080943 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:37.080909    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="102afe40-2887-4907-8ed8-c00b39339baa" path="/var/lib/kubelet/pods/102afe40-2887-4907-8ed8-c00b39339baa/volumes"
Sep 16 17:41:37.485248 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:37.485162    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg" podUID="0547c48a-584d-4972-84f2-53b616ff1075" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:41:47.484903 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:47.484846    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg" podUID="0547c48a-584d-4972-84f2-53b616ff1075" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:41:49.711465 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:49.711420    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg" podUID="0547c48a-584d-4972-84f2-53b616ff1075" containerName="main" containerID="cri-o://c3d2f01d5a517290f904d21423e37c39c90eacf659f19bd049e927a04291a1aa" gracePeriod=45
Sep 16 17:41:57.484798 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:57.484757    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg" podUID="0547c48a-584d-4972-84f2-53b616ff1075" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:41:57.485182 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:41:57.484905    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg"
Sep 16 17:42:07.485032 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:07.484994    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg" podUID="0547c48a-584d-4972-84f2-53b616ff1075" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:42:17.485215 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:17.485176    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg" podUID="0547c48a-584d-4972-84f2-53b616ff1075" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:42:19.920698 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:19.920661    2577 generic.go:356] "Generic (PLEG): container finished" podID="0547c48a-584d-4972-84f2-53b616ff1075" containerID="c3d2f01d5a517290f904d21423e37c39c90eacf659f19bd049e927a04291a1aa" exitCode=0
Sep 16 17:42:19.921058 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:19.920729    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg" event={"ID":"0547c48a-584d-4972-84f2-53b616ff1075","Type":"ContainerDied","Data":"c3d2f01d5a517290f904d21423e37c39c90eacf659f19bd049e927a04291a1aa"}
Sep 16 17:42:19.967013 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:19.966988    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg"
Sep 16 17:42:20.038713 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:20.038629    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0547c48a-584d-4972-84f2-53b616ff1075-tls-certs\") pod \"0547c48a-584d-4972-84f2-53b616ff1075\" (UID: \"0547c48a-584d-4972-84f2-53b616ff1075\") "
Sep 16 17:42:20.038713 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:20.038696    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fl69b\" (UniqueName: \"kubernetes.io/projected/0547c48a-584d-4972-84f2-53b616ff1075-kube-api-access-fl69b\") pod \"0547c48a-584d-4972-84f2-53b616ff1075\" (UID: \"0547c48a-584d-4972-84f2-53b616ff1075\") "
Sep 16 17:42:20.040692 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:20.040656    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0547c48a-584d-4972-84f2-53b616ff1075-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0547c48a-584d-4972-84f2-53b616ff1075" (UID: "0547c48a-584d-4972-84f2-53b616ff1075"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:42:20.040692 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:20.040686    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0547c48a-584d-4972-84f2-53b616ff1075-kube-api-access-fl69b" (OuterVolumeSpecName: "kube-api-access-fl69b") pod "0547c48a-584d-4972-84f2-53b616ff1075" (UID: "0547c48a-584d-4972-84f2-53b616ff1075"). InnerVolumeSpecName "kube-api-access-fl69b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:42:20.140159 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:20.140123    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0547c48a-584d-4972-84f2-53b616ff1075-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:42:20.140159 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:20.140153    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl69b\" (UniqueName: \"kubernetes.io/projected/0547c48a-584d-4972-84f2-53b616ff1075-kube-api-access-fl69b\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:42:20.927806 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:20.927768    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg" event={"ID":"0547c48a-584d-4972-84f2-53b616ff1075","Type":"ContainerDied","Data":"d03b511ac5585a670c01aba9eddda872c2f426c47b2de4d274269f23b8cd6a3c"}
Sep 16 17:42:20.927806 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:20.927789    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg"
Sep 16 17:42:20.928365 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:20.927831    2577 scope.go:117] "RemoveContainer" containerID="c3d2f01d5a517290f904d21423e37c39c90eacf659f19bd049e927a04291a1aa"
Sep 16 17:42:20.951896 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:20.951853    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg"]
Sep 16 17:42:20.956416 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:20.956392    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2tbmg"]
Sep 16 17:42:21.080960 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:21.080928    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0547c48a-584d-4972-84f2-53b616ff1075" path="/var/lib/kubelet/pods/0547c48a-584d-4972-84f2-53b616ff1075/volumes"
Sep 16 17:42:24.788477 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:24.788442    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:42:24.893906 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:24.893873    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:42:26.043680 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:26.043645    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:42:26.144788 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:26.144748    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:42:31.523516 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:31.523438    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:42:35.562015 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.561983    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk"]
Sep 16 17:42:35.562347 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.562337    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0547c48a-584d-4972-84f2-53b616ff1075" containerName="main"
Sep 16 17:42:35.562401 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.562348    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0547c48a-584d-4972-84f2-53b616ff1075" containerName="main"
Sep 16 17:42:35.562401 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.562368    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="102afe40-2887-4907-8ed8-c00b39339baa" containerName="main"
Sep 16 17:42:35.562401 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.562374    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="102afe40-2887-4907-8ed8-c00b39339baa" containerName="main"
Sep 16 17:42:35.562502 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.562422    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0547c48a-584d-4972-84f2-53b616ff1075" containerName="main"
Sep 16 17:42:35.562502 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.562432    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="102afe40-2887-4907-8ed8-c00b39339baa" containerName="main"
Sep 16 17:42:35.566801 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.566782    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk"
Sep 16 17:42:35.570822 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.570794    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 16 17:42:35.570992 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.570795    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 16 17:42:35.570992 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.570795    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 16 17:42:35.570992 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.570797    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-h4fxz\""
Sep 16 17:42:35.577979 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.577955    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk"]
Sep 16 17:42:35.584297 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.584257    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:42:35.672472 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.672435    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8550482-a4ea-4f1a-b225-1b74b78075a7-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-rpvsk\" (UID: \"e8550482-a4ea-4f1a-b225-1b74b78075a7\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk"
Sep 16 17:42:35.672653 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.672485    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8550482-a4ea-4f1a-b225-1b74b78075a7-home\") pod \"fc-concurrency-test-kserve-b5d44467c-rpvsk\" (UID: \"e8550482-a4ea-4f1a-b225-1b74b78075a7\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk"
Sep 16 17:42:35.672653 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.672562    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22lm4\" (UniqueName: \"kubernetes.io/projected/e8550482-a4ea-4f1a-b225-1b74b78075a7-kube-api-access-22lm4\") pod \"fc-concurrency-test-kserve-b5d44467c-rpvsk\" (UID: \"e8550482-a4ea-4f1a-b225-1b74b78075a7\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk"
Sep 16 17:42:35.672653 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.672581    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8550482-a4ea-4f1a-b225-1b74b78075a7-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-rpvsk\" (UID: \"e8550482-a4ea-4f1a-b225-1b74b78075a7\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk"
Sep 16 17:42:35.672653 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.672602    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8550482-a4ea-4f1a-b225-1b74b78075a7-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-rpvsk\" (UID: \"e8550482-a4ea-4f1a-b225-1b74b78075a7\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk"
Sep 16 17:42:35.672794 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.672672    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8550482-a4ea-4f1a-b225-1b74b78075a7-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-rpvsk\" (UID: \"e8550482-a4ea-4f1a-b225-1b74b78075a7\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk"
Sep 16 17:42:35.773804 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.773765    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8550482-a4ea-4f1a-b225-1b74b78075a7-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-rpvsk\" (UID: \"e8550482-a4ea-4f1a-b225-1b74b78075a7\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk"
Sep 16 17:42:35.774011 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.773810    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8550482-a4ea-4f1a-b225-1b74b78075a7-home\") pod \"fc-concurrency-test-kserve-b5d44467c-rpvsk\" (UID: \"e8550482-a4ea-4f1a-b225-1b74b78075a7\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk"
Sep 16 17:42:35.774011 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.773845    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22lm4\" (UniqueName: \"kubernetes.io/projected/e8550482-a4ea-4f1a-b225-1b74b78075a7-kube-api-access-22lm4\") pod \"fc-concurrency-test-kserve-b5d44467c-rpvsk\" (UID: \"e8550482-a4ea-4f1a-b225-1b74b78075a7\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk"
Sep 16 17:42:35.774011 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.773876    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8550482-a4ea-4f1a-b225-1b74b78075a7-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-rpvsk\" (UID: \"e8550482-a4ea-4f1a-b225-1b74b78075a7\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk"
Sep 16 17:42:35.774011 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.773905    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8550482-a4ea-4f1a-b225-1b74b78075a7-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-rpvsk\" (UID: \"e8550482-a4ea-4f1a-b225-1b74b78075a7\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk"
Sep 16 17:42:35.774011 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.773938    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8550482-a4ea-4f1a-b225-1b74b78075a7-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-rpvsk\" (UID: \"e8550482-a4ea-4f1a-b225-1b74b78075a7\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk"
Sep 16 17:42:35.774298 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.774278    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8550482-a4ea-4f1a-b225-1b74b78075a7-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-rpvsk\" (UID: \"e8550482-a4ea-4f1a-b225-1b74b78075a7\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk"
Sep 16 17:42:35.774335 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.774314    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8550482-a4ea-4f1a-b225-1b74b78075a7-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-rpvsk\" (UID: \"e8550482-a4ea-4f1a-b225-1b74b78075a7\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk"
Sep 16 17:42:35.774398 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.774363    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8550482-a4ea-4f1a-b225-1b74b78075a7-home\") pod \"fc-concurrency-test-kserve-b5d44467c-rpvsk\" (UID: \"e8550482-a4ea-4f1a-b225-1b74b78075a7\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk"
Sep 16 17:42:35.776361 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.776321    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8550482-a4ea-4f1a-b225-1b74b78075a7-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-rpvsk\" (UID: \"e8550482-a4ea-4f1a-b225-1b74b78075a7\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk"
Sep 16 17:42:35.776768 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.776744    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8550482-a4ea-4f1a-b225-1b74b78075a7-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-rpvsk\" (UID: \"e8550482-a4ea-4f1a-b225-1b74b78075a7\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk"
Sep 16 17:42:35.782619 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.782590    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22lm4\" (UniqueName: \"kubernetes.io/projected/e8550482-a4ea-4f1a-b225-1b74b78075a7-kube-api-access-22lm4\") pod \"fc-concurrency-test-kserve-b5d44467c-rpvsk\" (UID: \"e8550482-a4ea-4f1a-b225-1b74b78075a7\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk"
Sep 16 17:42:35.814735 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.814645    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp"]
Sep 16 17:42:35.820193 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.820166    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp"
Sep 16 17:42:35.823061 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.823035    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-q2qgp\""
Sep 16 17:42:35.825791 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.825764    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp"]
Sep 16 17:42:35.841159 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.841126    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:42:35.877555 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.877522    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk"
Sep 16 17:42:35.975598 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.975558    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5l5g\" (UniqueName: \"kubernetes.io/projected/e726b359-a6ba-4cd2-b2dd-67eb032fb70a-kube-api-access-r5l5g\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp\" (UID: \"e726b359-a6ba-4cd2-b2dd-67eb032fb70a\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp"
Sep 16 17:42:35.975753 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:35.975622    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e726b359-a6ba-4cd2-b2dd-67eb032fb70a-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp\" (UID: \"e726b359-a6ba-4cd2-b2dd-67eb032fb70a\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp"
Sep 16 17:42:36.076210 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:36.076112    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5l5g\" (UniqueName: \"kubernetes.io/projected/e726b359-a6ba-4cd2-b2dd-67eb032fb70a-kube-api-access-r5l5g\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp\" (UID: \"e726b359-a6ba-4cd2-b2dd-67eb032fb70a\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp"
Sep 16 17:42:36.076345 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:36.076249    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e726b359-a6ba-4cd2-b2dd-67eb032fb70a-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp\" (UID: \"e726b359-a6ba-4cd2-b2dd-67eb032fb70a\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp"
Sep 16 17:42:36.078844 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:36.078809    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e726b359-a6ba-4cd2-b2dd-67eb032fb70a-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp\" (UID: \"e726b359-a6ba-4cd2-b2dd-67eb032fb70a\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp"
Sep 16 17:42:36.084556 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:36.084530    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5l5g\" (UniqueName: \"kubernetes.io/projected/e726b359-a6ba-4cd2-b2dd-67eb032fb70a-kube-api-access-r5l5g\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp\" (UID: \"e726b359-a6ba-4cd2-b2dd-67eb032fb70a\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp"
Sep 16 17:42:36.132317 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:36.132284    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp"
Sep 16 17:42:36.151563 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:36.151536    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk"]
Sep 16 17:42:36.152639 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:42:36.152606    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8550482_a4ea_4f1a_b225_1b74b78075a7.slice/crio-06559ac02f92719ac5ce6837413c8c6fb6129ce7c838c709fc5ee0f572f71b2c WatchSource:0}: Error finding container 06559ac02f92719ac5ce6837413c8c6fb6129ce7c838c709fc5ee0f572f71b2c: Status 404 returned error can't find the container with id 06559ac02f92719ac5ce6837413c8c6fb6129ce7c838c709fc5ee0f572f71b2c
Sep 16 17:42:36.407825 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:36.407802    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp"]
Sep 16 17:42:36.409407 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:42:36.409368    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode726b359_a6ba_4cd2_b2dd_67eb032fb70a.slice/crio-cc44ab5be381abb7493f22c2eeed4aeb6fcecfa1cb6f550fa3b8c88092571f2e WatchSource:0}: Error finding container cc44ab5be381abb7493f22c2eeed4aeb6fcecfa1cb6f550fa3b8c88092571f2e: Status 404 returned error can't find the container with id cc44ab5be381abb7493f22c2eeed4aeb6fcecfa1cb6f550fa3b8c88092571f2e
Sep 16 17:42:37.014655 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:37.014616    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp" event={"ID":"e726b359-a6ba-4cd2-b2dd-67eb032fb70a","Type":"ContainerStarted","Data":"825ac79407a59df4f7b49f7d1a9e2c39737af1e924b8da784b927b46afa12fc7"}
Sep 16 17:42:37.015152 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:37.014687    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp"
Sep 16 17:42:37.015152 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:37.014709    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp" event={"ID":"e726b359-a6ba-4cd2-b2dd-67eb032fb70a","Type":"ContainerStarted","Data":"cc44ab5be381abb7493f22c2eeed4aeb6fcecfa1cb6f550fa3b8c88092571f2e"}
Sep 16 17:42:37.016422 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:37.016395    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk" event={"ID":"e8550482-a4ea-4f1a-b225-1b74b78075a7","Type":"ContainerStarted","Data":"abbe697ff687808c94d9e5517989e492e167a65f8db16027c4a3499c5c88f991"}
Sep 16 17:42:37.016552 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:37.016442    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk" event={"ID":"e8550482-a4ea-4f1a-b225-1b74b78075a7","Type":"ContainerStarted","Data":"06559ac02f92719ac5ce6837413c8c6fb6129ce7c838c709fc5ee0f572f71b2c"}
Sep 16 17:42:37.035076 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:37.035028    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp" podStartSLOduration=2.035011812 podStartE2EDuration="2.035011812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:42:37.032582325 +0000 UTC m=+1078.395261113" watchObservedRunningTime="2026-09-16 17:42:37.035011812 +0000 UTC m=+1078.397690615"
Sep 16 17:42:37.049492 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:37.049445    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk" podStartSLOduration=2.049428024 podStartE2EDuration="2.049428024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:42:37.049003296 +0000 UTC m=+1078.411682132" watchObservedRunningTime="2026-09-16 17:42:37.049428024 +0000 UTC m=+1078.412106810"
Sep 16 17:42:37.878399 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:37.878344    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk"
Sep 16 17:42:38.084302 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:38.084267    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:42:38.228659 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:38.228581    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:42:44.496953 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:44.496914    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:42:45.878443 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:45.878407    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk"
Sep 16 17:42:45.891096 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:45.891066    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk"
Sep 16 17:42:46.076368 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:42:46.076331    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk"
Sep 16 17:43:08.025030 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:08.024940    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp"
Sep 16 17:43:14.230362 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:14.230325    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk"]
Sep 16 17:43:14.233803 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:14.233771    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:43:14.235279 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:14.235252    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp"]
Sep 16 17:43:14.242027 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:43:14.241992    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 17:43:14.242027 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:43:14.242027 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 17:43:14.242027 ip-10-0-136-122 kubenswrapper[2577]:  > containerID="abbe697ff687808c94d9e5517989e492e167a65f8db16027c4a3499c5c88f991" cmd=["/bin/sleep","15"]
Sep 16 17:43:14.242305 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:14.242046    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk" message="" err=<
Sep 16 17:43:14.242305 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:43:14.242305 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 17:43:14.242305 ip-10-0-136-122 kubenswrapper[2577]:  >
Sep 16 17:43:14.242305 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:43:14.242060    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 17:43:14.242305 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:43:14.242305 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 17:43:14.242305 ip-10-0-136-122 kubenswrapper[2577]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk" podUID="e8550482-a4ea-4f1a-b225-1b74b78075a7" containerName="main" containerID="cri-o://abbe697ff687808c94d9e5517989e492e167a65f8db16027c4a3499c5c88f991"
Sep 16 17:43:14.242305 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:14.242093    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk" podUID="e8550482-a4ea-4f1a-b225-1b74b78075a7" containerName="main" containerID="cri-o://abbe697ff687808c94d9e5517989e492e167a65f8db16027c4a3499c5c88f991" gracePeriod=60
Sep 16 17:43:14.992626 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:14.992601    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk"
Sep 16 17:43:15.075504 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:15.075474    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22lm4\" (UniqueName: \"kubernetes.io/projected/e8550482-a4ea-4f1a-b225-1b74b78075a7-kube-api-access-22lm4\") pod \"e8550482-a4ea-4f1a-b225-1b74b78075a7\" (UID: \"e8550482-a4ea-4f1a-b225-1b74b78075a7\") "
Sep 16 17:43:15.075504 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:15.075503    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8550482-a4ea-4f1a-b225-1b74b78075a7-model-cache\") pod \"e8550482-a4ea-4f1a-b225-1b74b78075a7\" (UID: \"e8550482-a4ea-4f1a-b225-1b74b78075a7\") "
Sep 16 17:43:15.075740 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:15.075568    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8550482-a4ea-4f1a-b225-1b74b78075a7-dshm\") pod \"e8550482-a4ea-4f1a-b225-1b74b78075a7\" (UID: \"e8550482-a4ea-4f1a-b225-1b74b78075a7\") "
Sep 16 17:43:15.075740 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:15.075595    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8550482-a4ea-4f1a-b225-1b74b78075a7-tmp-dir\") pod \"e8550482-a4ea-4f1a-b225-1b74b78075a7\" (UID: \"e8550482-a4ea-4f1a-b225-1b74b78075a7\") "
Sep 16 17:43:15.075740 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:15.075621    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8550482-a4ea-4f1a-b225-1b74b78075a7-tls-certs\") pod \"e8550482-a4ea-4f1a-b225-1b74b78075a7\" (UID: \"e8550482-a4ea-4f1a-b225-1b74b78075a7\") "
Sep 16 17:43:15.075740 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:15.075675    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8550482-a4ea-4f1a-b225-1b74b78075a7-home\") pod \"e8550482-a4ea-4f1a-b225-1b74b78075a7\" (UID: \"e8550482-a4ea-4f1a-b225-1b74b78075a7\") "
Sep 16 17:43:15.075979 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:15.075856    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8550482-a4ea-4f1a-b225-1b74b78075a7-model-cache" (OuterVolumeSpecName: "model-cache") pod "e8550482-a4ea-4f1a-b225-1b74b78075a7" (UID: "e8550482-a4ea-4f1a-b225-1b74b78075a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:43:15.075979 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:15.075902    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8550482-a4ea-4f1a-b225-1b74b78075a7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e8550482-a4ea-4f1a-b225-1b74b78075a7" (UID: "e8550482-a4ea-4f1a-b225-1b74b78075a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:43:15.075979 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:15.075927    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8550482-a4ea-4f1a-b225-1b74b78075a7-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:43:15.076182 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:15.076041    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8550482-a4ea-4f1a-b225-1b74b78075a7-home" (OuterVolumeSpecName: "home") pod "e8550482-a4ea-4f1a-b225-1b74b78075a7" (UID: "e8550482-a4ea-4f1a-b225-1b74b78075a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:43:15.077852 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:15.077815    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8550482-a4ea-4f1a-b225-1b74b78075a7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e8550482-a4ea-4f1a-b225-1b74b78075a7" (UID: "e8550482-a4ea-4f1a-b225-1b74b78075a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:43:15.077989 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:15.077929    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8550482-a4ea-4f1a-b225-1b74b78075a7-kube-api-access-22lm4" (OuterVolumeSpecName: "kube-api-access-22lm4") pod "e8550482-a4ea-4f1a-b225-1b74b78075a7" (UID: "e8550482-a4ea-4f1a-b225-1b74b78075a7"). InnerVolumeSpecName "kube-api-access-22lm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:43:15.078053 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:15.078029    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8550482-a4ea-4f1a-b225-1b74b78075a7-dshm" (OuterVolumeSpecName: "dshm") pod "e8550482-a4ea-4f1a-b225-1b74b78075a7" (UID: "e8550482-a4ea-4f1a-b225-1b74b78075a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:43:15.177154 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:15.177065    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22lm4\" (UniqueName: \"kubernetes.io/projected/e8550482-a4ea-4f1a-b225-1b74b78075a7-kube-api-access-22lm4\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:43:15.177154 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:15.177093    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8550482-a4ea-4f1a-b225-1b74b78075a7-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:43:15.177154 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:15.177102    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8550482-a4ea-4f1a-b225-1b74b78075a7-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:43:15.177154 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:15.177112    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8550482-a4ea-4f1a-b225-1b74b78075a7-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:43:15.177154 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:15.177119    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8550482-a4ea-4f1a-b225-1b74b78075a7-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:43:15.222582 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:15.222548    2577 generic.go:356] "Generic (PLEG): container finished" podID="e8550482-a4ea-4f1a-b225-1b74b78075a7" containerID="abbe697ff687808c94d9e5517989e492e167a65f8db16027c4a3499c5c88f991" exitCode=0
Sep 16 17:43:15.222775 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:15.222597    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk"
Sep 16 17:43:15.222775 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:15.222639    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk" event={"ID":"e8550482-a4ea-4f1a-b225-1b74b78075a7","Type":"ContainerDied","Data":"abbe697ff687808c94d9e5517989e492e167a65f8db16027c4a3499c5c88f991"}
Sep 16 17:43:15.222775 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:15.222700    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk" event={"ID":"e8550482-a4ea-4f1a-b225-1b74b78075a7","Type":"ContainerDied","Data":"06559ac02f92719ac5ce6837413c8c6fb6129ce7c838c709fc5ee0f572f71b2c"}
Sep 16 17:43:15.222775 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:15.222731    2577 scope.go:117] "RemoveContainer" containerID="abbe697ff687808c94d9e5517989e492e167a65f8db16027c4a3499c5c88f991"
Sep 16 17:43:15.231922 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:15.231895    2577 scope.go:117] "RemoveContainer" containerID="abbe697ff687808c94d9e5517989e492e167a65f8db16027c4a3499c5c88f991"
Sep 16 17:43:15.232266 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:43:15.232242    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abbe697ff687808c94d9e5517989e492e167a65f8db16027c4a3499c5c88f991\": container with ID starting with abbe697ff687808c94d9e5517989e492e167a65f8db16027c4a3499c5c88f991 not found: ID does not exist" containerID="abbe697ff687808c94d9e5517989e492e167a65f8db16027c4a3499c5c88f991"
Sep 16 17:43:15.232314 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:15.232277    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abbe697ff687808c94d9e5517989e492e167a65f8db16027c4a3499c5c88f991"} err="failed to get container status \"abbe697ff687808c94d9e5517989e492e167a65f8db16027c4a3499c5c88f991\": rpc error: code = NotFound desc = could not find container \"abbe697ff687808c94d9e5517989e492e167a65f8db16027c4a3499c5c88f991\": container with ID starting with abbe697ff687808c94d9e5517989e492e167a65f8db16027c4a3499c5c88f991 not found: ID does not exist"
Sep 16 17:43:15.243247 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:15.243219    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk"]
Sep 16 17:43:15.247799 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:15.247775    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-rpvsk"]
Sep 16 17:43:15.585623 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:15.585584    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:43:17.081468 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:17.081432    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8550482-a4ea-4f1a-b225-1b74b78075a7" path="/var/lib/kubelet/pods/e8550482-a4ea-4f1a-b225-1b74b78075a7/volumes"
Sep 16 17:43:18.023996 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:18.023953    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp" podUID="e726b359-a6ba-4cd2-b2dd-67eb032fb70a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:43:28.024510 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:28.024470    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp" podUID="e726b359-a6ba-4cd2-b2dd-67eb032fb70a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:43:29.256348 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:29.256309    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp" podUID="e726b359-a6ba-4cd2-b2dd-67eb032fb70a" containerName="main" containerID="cri-o://825ac79407a59df4f7b49f7d1a9e2c39737af1e924b8da784b927b46afa12fc7" gracePeriod=45
Sep 16 17:43:37.927014 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:37.926980    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:43:38.024834 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:38.024795    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp" podUID="e726b359-a6ba-4cd2-b2dd-67eb032fb70a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:43:38.025018 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:38.024935    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp"
Sep 16 17:43:45.340365 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:45.340320    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:43:48.024583 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:48.024544    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp" podUID="e726b359-a6ba-4cd2-b2dd-67eb032fb70a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:43:58.024378 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:58.024335    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp" podUID="e726b359-a6ba-4cd2-b2dd-67eb032fb70a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:43:58.072312 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:58.072267    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:43:58.200917 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:58.200856    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:43:59.453807 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:59.453775    2577 generic.go:356] "Generic (PLEG): container finished" podID="e726b359-a6ba-4cd2-b2dd-67eb032fb70a" containerID="825ac79407a59df4f7b49f7d1a9e2c39737af1e924b8da784b927b46afa12fc7" exitCode=0
Sep 16 17:43:59.454178 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:59.453857    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp" event={"ID":"e726b359-a6ba-4cd2-b2dd-67eb032fb70a","Type":"ContainerDied","Data":"825ac79407a59df4f7b49f7d1a9e2c39737af1e924b8da784b927b46afa12fc7"}
Sep 16 17:43:59.506075 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:59.506050    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp"
Sep 16 17:43:59.554830 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:59.554748    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e726b359-a6ba-4cd2-b2dd-67eb032fb70a-tls-certs\") pod \"e726b359-a6ba-4cd2-b2dd-67eb032fb70a\" (UID: \"e726b359-a6ba-4cd2-b2dd-67eb032fb70a\") "
Sep 16 17:43:59.554830 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:59.554796    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5l5g\" (UniqueName: \"kubernetes.io/projected/e726b359-a6ba-4cd2-b2dd-67eb032fb70a-kube-api-access-r5l5g\") pod \"e726b359-a6ba-4cd2-b2dd-67eb032fb70a\" (UID: \"e726b359-a6ba-4cd2-b2dd-67eb032fb70a\") "
Sep 16 17:43:59.556829 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:59.556799    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e726b359-a6ba-4cd2-b2dd-67eb032fb70a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e726b359-a6ba-4cd2-b2dd-67eb032fb70a" (UID: "e726b359-a6ba-4cd2-b2dd-67eb032fb70a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:43:59.556964 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:59.556877    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e726b359-a6ba-4cd2-b2dd-67eb032fb70a-kube-api-access-r5l5g" (OuterVolumeSpecName: "kube-api-access-r5l5g") pod "e726b359-a6ba-4cd2-b2dd-67eb032fb70a" (UID: "e726b359-a6ba-4cd2-b2dd-67eb032fb70a"). InnerVolumeSpecName "kube-api-access-r5l5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:43:59.656098 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:59.656065    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e726b359-a6ba-4cd2-b2dd-67eb032fb70a-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:43:59.656098 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:43:59.656094    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5l5g\" (UniqueName: \"kubernetes.io/projected/e726b359-a6ba-4cd2-b2dd-67eb032fb70a-kube-api-access-r5l5g\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:44:00.460757 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:00.460724    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp"
Sep 16 17:44:00.460757 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:00.460737    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp" event={"ID":"e726b359-a6ba-4cd2-b2dd-67eb032fb70a","Type":"ContainerDied","Data":"cc44ab5be381abb7493f22c2eeed4aeb6fcecfa1cb6f550fa3b8c88092571f2e"}
Sep 16 17:44:00.461199 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:00.460799    2577 scope.go:117] "RemoveContainer" containerID="825ac79407a59df4f7b49f7d1a9e2c39737af1e924b8da784b927b46afa12fc7"
Sep 16 17:44:00.484368 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:00.484341    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp"]
Sep 16 17:44:00.491626 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:00.491598    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-xcqkp"]
Sep 16 17:44:01.081478 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:01.081442    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e726b359-a6ba-4cd2-b2dd-67eb032fb70a" path="/var/lib/kubelet/pods/e726b359-a6ba-4cd2-b2dd-67eb032fb70a/volumes"
Sep 16 17:44:03.291210 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.291171    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:44:03.500518 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.500485    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:44:03.620513 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.620483    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:44:03.652675 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.652640    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 17:44:03.653047 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.653033    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e726b359-a6ba-4cd2-b2dd-67eb032fb70a" containerName="main"
Sep 16 17:44:03.653110 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.653049    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e726b359-a6ba-4cd2-b2dd-67eb032fb70a" containerName="main"
Sep 16 17:44:03.653110 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.653060    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8550482-a4ea-4f1a-b225-1b74b78075a7" containerName="main"
Sep 16 17:44:03.653110 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.653066    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8550482-a4ea-4f1a-b225-1b74b78075a7" containerName="main"
Sep 16 17:44:03.653215 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.653145    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e726b359-a6ba-4cd2-b2dd-67eb032fb70a" containerName="main"
Sep 16 17:44:03.653215 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.653153    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8550482-a4ea-4f1a-b225-1b74b78075a7" containerName="main"
Sep 16 17:44:03.657614 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.657595    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 17:44:03.662052 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.662032    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 16 17:44:03.662593 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.662578    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 16 17:44:03.662641 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.662602    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 16 17:44:03.663248 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.663223    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-fzqtd\""
Sep 16 17:44:03.663347 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.663255    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 17:44:03.663347 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.663285    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 16 17:44:03.663446 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.663378    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-cmk64\""
Sep 16 17:44:03.683453 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.683429    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 17:44:03.690317 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.690294    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7l9j\" (UniqueName: \"kubernetes.io/projected/178b8d32-bf69-4588-9400-9afaa06ee302-kube-api-access-m7l9j\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"178b8d32-bf69-4588-9400-9afaa06ee302\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 17:44:03.690436 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.690331    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/178b8d32-bf69-4588-9400-9afaa06ee302-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"178b8d32-bf69-4588-9400-9afaa06ee302\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 17:44:03.690436 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.690360    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/178b8d32-bf69-4588-9400-9afaa06ee302-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"178b8d32-bf69-4588-9400-9afaa06ee302\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 17:44:03.690528 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.690428    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/178b8d32-bf69-4588-9400-9afaa06ee302-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"178b8d32-bf69-4588-9400-9afaa06ee302\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 17:44:03.690528 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.690463    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/178b8d32-bf69-4588-9400-9afaa06ee302-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"178b8d32-bf69-4588-9400-9afaa06ee302\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 17:44:03.690528 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.690482    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/178b8d32-bf69-4588-9400-9afaa06ee302-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"178b8d32-bf69-4588-9400-9afaa06ee302\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 17:44:03.690528 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.690501    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/178b8d32-bf69-4588-9400-9afaa06ee302-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"178b8d32-bf69-4588-9400-9afaa06ee302\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 17:44:03.690697 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.690554    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/178b8d32-bf69-4588-9400-9afaa06ee302-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"178b8d32-bf69-4588-9400-9afaa06ee302\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 17:44:03.792018 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.791979    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7l9j\" (UniqueName: \"kubernetes.io/projected/178b8d32-bf69-4588-9400-9afaa06ee302-kube-api-access-m7l9j\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"178b8d32-bf69-4588-9400-9afaa06ee302\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 17:44:03.792186 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.792043    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/178b8d32-bf69-4588-9400-9afaa06ee302-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"178b8d32-bf69-4588-9400-9afaa06ee302\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 17:44:03.792186 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.792089    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/178b8d32-bf69-4588-9400-9afaa06ee302-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"178b8d32-bf69-4588-9400-9afaa06ee302\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 17:44:03.792186 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.792116    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/178b8d32-bf69-4588-9400-9afaa06ee302-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"178b8d32-bf69-4588-9400-9afaa06ee302\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 17:44:03.792186 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.792148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/178b8d32-bf69-4588-9400-9afaa06ee302-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"178b8d32-bf69-4588-9400-9afaa06ee302\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 17:44:03.792186 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.792172    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/178b8d32-bf69-4588-9400-9afaa06ee302-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"178b8d32-bf69-4588-9400-9afaa06ee302\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 17:44:03.792461 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.792201    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/178b8d32-bf69-4588-9400-9afaa06ee302-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"178b8d32-bf69-4588-9400-9afaa06ee302\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 17:44:03.792461 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.792230    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/178b8d32-bf69-4588-9400-9afaa06ee302-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"178b8d32-bf69-4588-9400-9afaa06ee302\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 17:44:03.792579 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.792465    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/178b8d32-bf69-4588-9400-9afaa06ee302-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"178b8d32-bf69-4588-9400-9afaa06ee302\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 17:44:03.792811 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.792784    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/178b8d32-bf69-4588-9400-9afaa06ee302-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"178b8d32-bf69-4588-9400-9afaa06ee302\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 17:44:03.793435 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.792826    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/178b8d32-bf69-4588-9400-9afaa06ee302-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"178b8d32-bf69-4588-9400-9afaa06ee302\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 17:44:03.793435 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.793106    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/178b8d32-bf69-4588-9400-9afaa06ee302-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"178b8d32-bf69-4588-9400-9afaa06ee302\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 17:44:03.793435 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.793367    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/178b8d32-bf69-4588-9400-9afaa06ee302-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"178b8d32-bf69-4588-9400-9afaa06ee302\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 17:44:03.794933 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.794909    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/178b8d32-bf69-4588-9400-9afaa06ee302-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"178b8d32-bf69-4588-9400-9afaa06ee302\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 17:44:03.795228 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.795205    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/178b8d32-bf69-4588-9400-9afaa06ee302-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"178b8d32-bf69-4588-9400-9afaa06ee302\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 17:44:03.814523 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.814493    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2"]
Sep 16 17:44:03.817793 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.817773    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7l9j\" (UniqueName: \"kubernetes.io/projected/178b8d32-bf69-4588-9400-9afaa06ee302-kube-api-access-m7l9j\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"178b8d32-bf69-4588-9400-9afaa06ee302\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 17:44:03.818976 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.818962    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2"
Sep 16 17:44:03.824012 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.823993    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-f5l26\""
Sep 16 17:44:03.841386 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.841360    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2"]
Sep 16 17:44:03.892431 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.892348    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:44:03.892676 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.892657    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b9487d-8964-4dc8-956b-2ec1c59a6d67-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2\" (UID: \"33b9487d-8964-4dc8-956b-2ec1c59a6d67\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2"
Sep 16 17:44:03.892728 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.892715    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n46rv\" (UniqueName: \"kubernetes.io/projected/33b9487d-8964-4dc8-956b-2ec1c59a6d67-kube-api-access-n46rv\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2\" (UID: \"33b9487d-8964-4dc8-956b-2ec1c59a6d67\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2"
Sep 16 17:44:03.968229 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.968200    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 17:44:03.993582 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.993548    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b9487d-8964-4dc8-956b-2ec1c59a6d67-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2\" (UID: \"33b9487d-8964-4dc8-956b-2ec1c59a6d67\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2"
Sep 16 17:44:03.993717 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.993642    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n46rv\" (UniqueName: \"kubernetes.io/projected/33b9487d-8964-4dc8-956b-2ec1c59a6d67-kube-api-access-n46rv\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2\" (UID: \"33b9487d-8964-4dc8-956b-2ec1c59a6d67\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2"
Sep 16 17:44:03.996273 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:03.996245    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b9487d-8964-4dc8-956b-2ec1c59a6d67-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2\" (UID: \"33b9487d-8964-4dc8-956b-2ec1c59a6d67\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2"
Sep 16 17:44:04.003747 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:04.003715    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n46rv\" (UniqueName: \"kubernetes.io/projected/33b9487d-8964-4dc8-956b-2ec1c59a6d67-kube-api-access-n46rv\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2\" (UID: \"33b9487d-8964-4dc8-956b-2ec1c59a6d67\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2"
Sep 16 17:44:04.133480 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:04.133442    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2"
Sep 16 17:44:04.249205 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:04.249180    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 17:44:04.250017 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:44:04.249992    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod178b8d32_bf69_4588_9400_9afaa06ee302.slice/crio-c2e00d705fc9213d58bbbfc2a8745592d6eac64c871b7ae1b93b301f896d67dc WatchSource:0}: Error finding container c2e00d705fc9213d58bbbfc2a8745592d6eac64c871b7ae1b93b301f896d67dc: Status 404 returned error can't find the container with id c2e00d705fc9213d58bbbfc2a8745592d6eac64c871b7ae1b93b301f896d67dc
Sep 16 17:44:04.251323 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:04.251293    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:44:04.391759 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:04.391709    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:44:04.411489 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:04.411461    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2"]
Sep 16 17:44:04.412136 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:44:04.412112    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b9487d_8964_4dc8_956b_2ec1c59a6d67.slice/crio-300a87b3bacd22baf5d7ae612868f3114dbe870c9884f83c53e66c0c6d521de1 WatchSource:0}: Error finding container 300a87b3bacd22baf5d7ae612868f3114dbe870c9884f83c53e66c0c6d521de1: Status 404 returned error can't find the container with id 300a87b3bacd22baf5d7ae612868f3114dbe870c9884f83c53e66c0c6d521de1
Sep 16 17:44:04.484787 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:04.484756    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2" event={"ID":"33b9487d-8964-4dc8-956b-2ec1c59a6d67","Type":"ContainerStarted","Data":"1644ce23ea9b04d74592e9befde9c87e973fe644930823d84d07aa8d214ff816"}
Sep 16 17:44:04.485104 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:04.484818    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2" event={"ID":"33b9487d-8964-4dc8-956b-2ec1c59a6d67","Type":"ContainerStarted","Data":"300a87b3bacd22baf5d7ae612868f3114dbe870c9884f83c53e66c0c6d521de1"}
Sep 16 17:44:04.485104 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:04.484973    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2"
Sep 16 17:44:04.486726 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:04.486698    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"178b8d32-bf69-4588-9400-9afaa06ee302","Type":"ContainerStarted","Data":"8137640ad928efa52635a47882e67a7d6b6fab895a6dd832db276bc402e01369"}
Sep 16 17:44:04.486843 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:04.486734    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"178b8d32-bf69-4588-9400-9afaa06ee302","Type":"ContainerStarted","Data":"c2e00d705fc9213d58bbbfc2a8745592d6eac64c871b7ae1b93b301f896d67dc"}
Sep 16 17:44:04.496261 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:04.496197    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:44:04.515973 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:04.515923    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2" podStartSLOduration=1.515906191 podStartE2EDuration="1.515906191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:44:04.513058653 +0000 UTC m=+1165.875737454" watchObservedRunningTime="2026-09-16 17:44:04.515906191 +0000 UTC m=+1165.878584976"
Sep 16 17:44:04.759284 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:04.759243    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:44:04.893136 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:04.893094    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:44:05.932468 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:05.932415    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:44:07.508371 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:07.508342    2577 generic.go:356] "Generic (PLEG): container finished" podID="178b8d32-bf69-4588-9400-9afaa06ee302" containerID="8137640ad928efa52635a47882e67a7d6b6fab895a6dd832db276bc402e01369" exitCode=0
Sep 16 17:44:07.508734 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:07.508411    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"178b8d32-bf69-4588-9400-9afaa06ee302","Type":"ContainerDied","Data":"8137640ad928efa52635a47882e67a7d6b6fab895a6dd832db276bc402e01369"}
Sep 16 17:44:07.509995 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:07.509980    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 17:44:08.516978 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:08.516767    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:44:11.280762 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:11.280716    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:44:11.987166 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:11.987101    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:44:12.089030 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:12.088977    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:44:16.717255 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:16.717223    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:44:17.968219 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:17.968187    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:44:18.053482 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:18.053445    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:44:24.772557 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:24.772522    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:44:35.496299 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:35.496266    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2"
Sep 16 17:44:36.716183 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:36.716135    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"178b8d32-bf69-4588-9400-9afaa06ee302","Type":"ContainerStarted","Data":"a1513a649bae7ac42a588f1aa8d60b0986aa9779025c2fc6332d84ef65508706"}
Sep 16 17:44:36.747155 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:36.747090    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.928097857 podStartE2EDuration="33.747068173s" totalImagesPullingTime="28.522613303s" totalInitContainerRuntime="2.296357013s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:44:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:44:36.744123621 +0000 UTC m=+1198.106802407" watchObservedRunningTime="2026-09-16 17:44:36.747068173 +0000 UTC m=+1198.109746960"
Sep 16 17:44:36.969142 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:36.969054    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 17:44:40.759991 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:40.759953    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n"]
Sep 16 17:44:40.816809 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:40.816778    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:44:40.875880 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:40.875832    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n"]
Sep 16 17:44:40.876057 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:40.875946    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n"
Sep 16 17:44:40.884652 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:40.884517    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 16 17:44:40.884881 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:40.884842    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-h69dk\""
Sep 16 17:44:40.885045 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:40.884908    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Sep 16 17:44:40.885045 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:40.884911    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 16 17:44:41.043708 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:41.043616    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n\" (UID: \"a9638d7e-b64f-4976-89f3-8e2ace01b1e0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n"
Sep 16 17:44:41.043708 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:41.043668    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n\" (UID: \"a9638d7e-b64f-4976-89f3-8e2ace01b1e0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n"
Sep 16 17:44:41.043708 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:41.043704    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggfxv\" (UniqueName: \"kubernetes.io/projected/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-kube-api-access-ggfxv\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n\" (UID: \"a9638d7e-b64f-4976-89f3-8e2ace01b1e0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n"
Sep 16 17:44:41.044033 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:41.043806    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n\" (UID: \"a9638d7e-b64f-4976-89f3-8e2ace01b1e0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n"
Sep 16 17:44:41.044033 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:41.043896    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n\" (UID: \"a9638d7e-b64f-4976-89f3-8e2ace01b1e0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n"
Sep 16 17:44:41.044033 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:41.043942    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n\" (UID: \"a9638d7e-b64f-4976-89f3-8e2ace01b1e0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n"
Sep 16 17:44:41.144890 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:41.144834    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n\" (UID: \"a9638d7e-b64f-4976-89f3-8e2ace01b1e0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n"
Sep 16 17:44:41.145080 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:41.144903    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n\" (UID: \"a9638d7e-b64f-4976-89f3-8e2ace01b1e0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n"
Sep 16 17:44:41.145080 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:41.144965    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n\" (UID: \"a9638d7e-b64f-4976-89f3-8e2ace01b1e0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n"
Sep 16 17:44:41.145080 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:41.145007    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n\" (UID: \"a9638d7e-b64f-4976-89f3-8e2ace01b1e0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n"
Sep 16 17:44:41.145080 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:41.145035    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggfxv\" (UniqueName: \"kubernetes.io/projected/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-kube-api-access-ggfxv\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n\" (UID: \"a9638d7e-b64f-4976-89f3-8e2ace01b1e0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n"
Sep 16 17:44:41.145308 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:41.145111    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n\" (UID: \"a9638d7e-b64f-4976-89f3-8e2ace01b1e0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n"
Sep 16 17:44:41.145456 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:41.145401    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n\" (UID: \"a9638d7e-b64f-4976-89f3-8e2ace01b1e0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n"
Sep 16 17:44:41.145554 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:41.145518    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n\" (UID: \"a9638d7e-b64f-4976-89f3-8e2ace01b1e0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n"
Sep 16 17:44:41.145618 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:41.145565    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n\" (UID: \"a9638d7e-b64f-4976-89f3-8e2ace01b1e0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n"
Sep 16 17:44:41.147558 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:41.147533    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n\" (UID: \"a9638d7e-b64f-4976-89f3-8e2ace01b1e0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n"
Sep 16 17:44:41.147656 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:41.147639    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n\" (UID: \"a9638d7e-b64f-4976-89f3-8e2ace01b1e0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n"
Sep 16 17:44:41.159896 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:41.159847    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggfxv\" (UniqueName: \"kubernetes.io/projected/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-kube-api-access-ggfxv\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n\" (UID: \"a9638d7e-b64f-4976-89f3-8e2ace01b1e0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n"
Sep 16 17:44:41.189507 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:41.189467    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n"
Sep 16 17:44:41.490583 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:41.490551    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n"]
Sep 16 17:44:41.492441 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:44:41.492403    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9638d7e_b64f_4976_89f3_8e2ace01b1e0.slice/crio-80a53b66fbcc1d3d9f047e76168dd981416bcb268430cc5a4dc8872320d7dedd WatchSource:0}: Error finding container 80a53b66fbcc1d3d9f047e76168dd981416bcb268430cc5a4dc8872320d7dedd: Status 404 returned error can't find the container with id 80a53b66fbcc1d3d9f047e76168dd981416bcb268430cc5a4dc8872320d7dedd
Sep 16 17:44:41.751224 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:41.751131    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n" event={"ID":"a9638d7e-b64f-4976-89f3-8e2ace01b1e0","Type":"ContainerStarted","Data":"49235bfa0ba17957481237cb727c8649269c7ccb4ae71a22e53d4610f5b5e4ee"}
Sep 16 17:44:41.751408 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:41.751236    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n" event={"ID":"a9638d7e-b64f-4976-89f3-8e2ace01b1e0","Type":"ContainerStarted","Data":"80a53b66fbcc1d3d9f047e76168dd981416bcb268430cc5a4dc8872320d7dedd"}
Sep 16 17:44:41.771407 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:41.771350    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n" podStartSLOduration=1.771336392 podStartE2EDuration="1.771336392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:44:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:44:41.770605151 +0000 UTC m=+1203.133283943" watchObservedRunningTime="2026-09-16 17:44:41.771336392 +0000 UTC m=+1203.134015177"
Sep 16 17:44:42.190359 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:42.190318    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n"
Sep 16 17:44:43.281601 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:43.281571    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:44:43.382924 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:43.382888    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:44:43.485508 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:43.485472    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:44:43.606678 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:43.606640    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:44:43.708092 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:43.708059    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:44:43.968483 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:43.968396    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 17:44:43.970414 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:43.970390    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="178b8d32-bf69-4588-9400-9afaa06ee302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 16 17:44:46.644224 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:46.644182    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n"]
Sep 16 17:44:46.654017 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:44:46.653977    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 17:44:46.654017 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:44:46.654017 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 17:44:46.654017 ip-10-0-136-122 kubenswrapper[2577]:  > containerID="49235bfa0ba17957481237cb727c8649269c7ccb4ae71a22e53d4610f5b5e4ee" cmd=["/bin/sleep","15"]
Sep 16 17:44:46.654282 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:46.654031    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n" message="" err=<
Sep 16 17:44:46.654282 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:44:46.654282 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 17:44:46.654282 ip-10-0-136-122 kubenswrapper[2577]:  >
Sep 16 17:44:46.654282 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:44:46.654045    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 17:44:46.654282 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:44:46.654282 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 17:44:46.654282 ip-10-0-136-122 kubenswrapper[2577]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n" podUID="a9638d7e-b64f-4976-89f3-8e2ace01b1e0" containerName="main" containerID="cri-o://49235bfa0ba17957481237cb727c8649269c7ccb4ae71a22e53d4610f5b5e4ee"
Sep 16 17:44:46.654282 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:46.654087    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n" podUID="a9638d7e-b64f-4976-89f3-8e2ace01b1e0" containerName="main" containerID="cri-o://49235bfa0ba17957481237cb727c8649269c7ccb4ae71a22e53d4610f5b5e4ee" gracePeriod=60
Sep 16 17:44:46.681625 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:46.681591    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:44:47.000023 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:46.999985    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:44:47.462050 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:47.462012    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n"
Sep 16 17:44:47.499109 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:47.499080    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-tmp-dir\") pod \"a9638d7e-b64f-4976-89f3-8e2ace01b1e0\" (UID: \"a9638d7e-b64f-4976-89f3-8e2ace01b1e0\") "
Sep 16 17:44:47.499288 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:47.499114    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-tls-certs\") pod \"a9638d7e-b64f-4976-89f3-8e2ace01b1e0\" (UID: \"a9638d7e-b64f-4976-89f3-8e2ace01b1e0\") "
Sep 16 17:44:47.499288 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:47.499149    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-dshm\") pod \"a9638d7e-b64f-4976-89f3-8e2ace01b1e0\" (UID: \"a9638d7e-b64f-4976-89f3-8e2ace01b1e0\") "
Sep 16 17:44:47.499288 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:47.499271    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-model-cache\") pod \"a9638d7e-b64f-4976-89f3-8e2ace01b1e0\" (UID: \"a9638d7e-b64f-4976-89f3-8e2ace01b1e0\") "
Sep 16 17:44:47.499464 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:47.499336    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggfxv\" (UniqueName: \"kubernetes.io/projected/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-kube-api-access-ggfxv\") pod \"a9638d7e-b64f-4976-89f3-8e2ace01b1e0\" (UID: \"a9638d7e-b64f-4976-89f3-8e2ace01b1e0\") "
Sep 16 17:44:47.499464 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:47.499362    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a9638d7e-b64f-4976-89f3-8e2ace01b1e0" (UID: "a9638d7e-b64f-4976-89f3-8e2ace01b1e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:44:47.499464 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:47.499395    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-home\") pod \"a9638d7e-b64f-4976-89f3-8e2ace01b1e0\" (UID: \"a9638d7e-b64f-4976-89f3-8e2ace01b1e0\") "
Sep 16 17:44:47.499621 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:47.499500    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-model-cache" (OuterVolumeSpecName: "model-cache") pod "a9638d7e-b64f-4976-89f3-8e2ace01b1e0" (UID: "a9638d7e-b64f-4976-89f3-8e2ace01b1e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:44:47.499700 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:47.499678    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-home" (OuterVolumeSpecName: "home") pod "a9638d7e-b64f-4976-89f3-8e2ace01b1e0" (UID: "a9638d7e-b64f-4976-89f3-8e2ace01b1e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:44:47.499759 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:47.499703    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:44:47.499759 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:47.499724    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:44:47.501532 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:47.501501    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-dshm" (OuterVolumeSpecName: "dshm") pod "a9638d7e-b64f-4976-89f3-8e2ace01b1e0" (UID: "a9638d7e-b64f-4976-89f3-8e2ace01b1e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:44:47.501856 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:47.501828    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-kube-api-access-ggfxv" (OuterVolumeSpecName: "kube-api-access-ggfxv") pod "a9638d7e-b64f-4976-89f3-8e2ace01b1e0" (UID: "a9638d7e-b64f-4976-89f3-8e2ace01b1e0"). InnerVolumeSpecName "kube-api-access-ggfxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:44:47.501978 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:47.501835    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a9638d7e-b64f-4976-89f3-8e2ace01b1e0" (UID: "a9638d7e-b64f-4976-89f3-8e2ace01b1e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:44:47.600717 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:47.600666    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:44:47.600717 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:47.600717    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:44:47.601045 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:47.600734    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggfxv\" (UniqueName: \"kubernetes.io/projected/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-kube-api-access-ggfxv\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:44:47.601045 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:47.600750    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9638d7e-b64f-4976-89f3-8e2ace01b1e0-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:44:47.788299 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:47.788267    2577 generic.go:356] "Generic (PLEG): container finished" podID="a9638d7e-b64f-4976-89f3-8e2ace01b1e0" containerID="49235bfa0ba17957481237cb727c8649269c7ccb4ae71a22e53d4610f5b5e4ee" exitCode=0
Sep 16 17:44:47.788688 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:47.788309    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n"
Sep 16 17:44:47.788688 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:47.788351    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n" event={"ID":"a9638d7e-b64f-4976-89f3-8e2ace01b1e0","Type":"ContainerDied","Data":"49235bfa0ba17957481237cb727c8649269c7ccb4ae71a22e53d4610f5b5e4ee"}
Sep 16 17:44:47.788688 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:47.788410    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n" event={"ID":"a9638d7e-b64f-4976-89f3-8e2ace01b1e0","Type":"ContainerDied","Data":"80a53b66fbcc1d3d9f047e76168dd981416bcb268430cc5a4dc8872320d7dedd"}
Sep 16 17:44:47.788688 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:47.788438    2577 scope.go:117] "RemoveContainer" containerID="49235bfa0ba17957481237cb727c8649269c7ccb4ae71a22e53d4610f5b5e4ee"
Sep 16 17:44:47.798619 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:47.798601    2577 scope.go:117] "RemoveContainer" containerID="49235bfa0ba17957481237cb727c8649269c7ccb4ae71a22e53d4610f5b5e4ee"
Sep 16 17:44:47.798918 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:44:47.798890    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49235bfa0ba17957481237cb727c8649269c7ccb4ae71a22e53d4610f5b5e4ee\": container with ID starting with 49235bfa0ba17957481237cb727c8649269c7ccb4ae71a22e53d4610f5b5e4ee not found: ID does not exist" containerID="49235bfa0ba17957481237cb727c8649269c7ccb4ae71a22e53d4610f5b5e4ee"
Sep 16 17:44:47.798995 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:47.798930    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49235bfa0ba17957481237cb727c8649269c7ccb4ae71a22e53d4610f5b5e4ee"} err="failed to get container status \"49235bfa0ba17957481237cb727c8649269c7ccb4ae71a22e53d4610f5b5e4ee\": rpc error: code = NotFound desc = could not find container \"49235bfa0ba17957481237cb727c8649269c7ccb4ae71a22e53d4610f5b5e4ee\": container with ID starting with 49235bfa0ba17957481237cb727c8649269c7ccb4ae71a22e53d4610f5b5e4ee not found: ID does not exist"
Sep 16 17:44:47.813847 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:47.813821    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n"]
Sep 16 17:44:47.816018 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:47.815999    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-8vd6n"]
Sep 16 17:44:48.306705 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:48.306669    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:44:49.082623 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:49.082590    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9638d7e-b64f-4976-89f3-8e2ace01b1e0" path="/var/lib/kubelet/pods/a9638d7e-b64f-4976-89f3-8e2ace01b1e0/volumes"
Sep 16 17:44:53.851367 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:53.851326    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:44:53.969022 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:53.968973    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="178b8d32-bf69-4588-9400-9afaa06ee302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 16 17:44:54.461715 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:44:54.461668    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:45:03.360502 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:03.360462    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8"]
Sep 16 17:45:03.361072 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:03.361052    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9638d7e-b64f-4976-89f3-8e2ace01b1e0" containerName="main"
Sep 16 17:45:03.361126 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:03.361076    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9638d7e-b64f-4976-89f3-8e2ace01b1e0" containerName="main"
Sep 16 17:45:03.361188 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:03.361175    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9638d7e-b64f-4976-89f3-8e2ace01b1e0" containerName="main"
Sep 16 17:45:03.366106 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:03.366086    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8"
Sep 16 17:45:03.370035 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:03.370009    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 16 17:45:03.370173 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:03.370065    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-rdrk8\""
Sep 16 17:45:03.370173 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:03.370074    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 16 17:45:03.370173 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:03.370127    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Sep 16 17:45:03.376261 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:03.376238    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8"]
Sep 16 17:45:03.433109 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:03.433063    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5bda94e-3f98-4e8d-a96b-326290b6d381-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8\" (UID: \"a5bda94e-3f98-4e8d-a96b-326290b6d381\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8"
Sep 16 17:45:03.433289 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:03.433120    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5bda94e-3f98-4e8d-a96b-326290b6d381-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8\" (UID: \"a5bda94e-3f98-4e8d-a96b-326290b6d381\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8"
Sep 16 17:45:03.433289 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:03.433216    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5bda94e-3f98-4e8d-a96b-326290b6d381-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8\" (UID: \"a5bda94e-3f98-4e8d-a96b-326290b6d381\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8"
Sep 16 17:45:03.433397 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:03.433288    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z9n8\" (UniqueName: \"kubernetes.io/projected/a5bda94e-3f98-4e8d-a96b-326290b6d381-kube-api-access-6z9n8\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8\" (UID: \"a5bda94e-3f98-4e8d-a96b-326290b6d381\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8"
Sep 16 17:45:03.433397 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:03.433338    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5bda94e-3f98-4e8d-a96b-326290b6d381-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8\" (UID: \"a5bda94e-3f98-4e8d-a96b-326290b6d381\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8"
Sep 16 17:45:03.433501 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:03.433425    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5bda94e-3f98-4e8d-a96b-326290b6d381-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8\" (UID: \"a5bda94e-3f98-4e8d-a96b-326290b6d381\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8"
Sep 16 17:45:03.447763 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:03.447717    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:45:03.533917 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:03.533883    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5bda94e-3f98-4e8d-a96b-326290b6d381-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8\" (UID: \"a5bda94e-3f98-4e8d-a96b-326290b6d381\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8"
Sep 16 17:45:03.534093 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:03.533930    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5bda94e-3f98-4e8d-a96b-326290b6d381-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8\" (UID: \"a5bda94e-3f98-4e8d-a96b-326290b6d381\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8"
Sep 16 17:45:03.534093 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:03.533961    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z9n8\" (UniqueName: \"kubernetes.io/projected/a5bda94e-3f98-4e8d-a96b-326290b6d381-kube-api-access-6z9n8\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8\" (UID: \"a5bda94e-3f98-4e8d-a96b-326290b6d381\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8"
Sep 16 17:45:03.534093 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:03.533986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5bda94e-3f98-4e8d-a96b-326290b6d381-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8\" (UID: \"a5bda94e-3f98-4e8d-a96b-326290b6d381\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8"
Sep 16 17:45:03.534093 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:03.534016    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5bda94e-3f98-4e8d-a96b-326290b6d381-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8\" (UID: \"a5bda94e-3f98-4e8d-a96b-326290b6d381\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8"
Sep 16 17:45:03.534093 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:03.534070    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5bda94e-3f98-4e8d-a96b-326290b6d381-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8\" (UID: \"a5bda94e-3f98-4e8d-a96b-326290b6d381\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8"
Sep 16 17:45:03.534571 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:03.534547    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5bda94e-3f98-4e8d-a96b-326290b6d381-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8\" (UID: \"a5bda94e-3f98-4e8d-a96b-326290b6d381\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8"
Sep 16 17:45:03.534571 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:03.534556    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5bda94e-3f98-4e8d-a96b-326290b6d381-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8\" (UID: \"a5bda94e-3f98-4e8d-a96b-326290b6d381\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8"
Sep 16 17:45:03.534571 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:03.534570    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5bda94e-3f98-4e8d-a96b-326290b6d381-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8\" (UID: \"a5bda94e-3f98-4e8d-a96b-326290b6d381\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8"
Sep 16 17:45:03.536406 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:03.536374    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5bda94e-3f98-4e8d-a96b-326290b6d381-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8\" (UID: \"a5bda94e-3f98-4e8d-a96b-326290b6d381\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8"
Sep 16 17:45:03.536642 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:03.536627    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5bda94e-3f98-4e8d-a96b-326290b6d381-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8\" (UID: \"a5bda94e-3f98-4e8d-a96b-326290b6d381\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8"
Sep 16 17:45:03.542106 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:03.542085    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z9n8\" (UniqueName: \"kubernetes.io/projected/a5bda94e-3f98-4e8d-a96b-326290b6d381-kube-api-access-6z9n8\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8\" (UID: \"a5bda94e-3f98-4e8d-a96b-326290b6d381\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8"
Sep 16 17:45:03.677684 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:03.677584    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8"
Sep 16 17:45:03.969736 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:03.969586    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="178b8d32-bf69-4588-9400-9afaa06ee302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 16 17:45:03.974000 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:03.973974    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8"]
Sep 16 17:45:03.975399 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:45:03.975376    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5bda94e_3f98_4e8d_a96b_326290b6d381.slice/crio-358fa3924ba216b303363bedceb73e272047f9af707bbd156b1d0ac51bb1cfd3 WatchSource:0}: Error finding container 358fa3924ba216b303363bedceb73e272047f9af707bbd156b1d0ac51bb1cfd3: Status 404 returned error can't find the container with id 358fa3924ba216b303363bedceb73e272047f9af707bbd156b1d0ac51bb1cfd3
Sep 16 17:45:04.887048 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:04.887012    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8" event={"ID":"a5bda94e-3f98-4e8d-a96b-326290b6d381","Type":"ContainerStarted","Data":"449d51b6d47cc03c4e9b6a14331eadce7f54582cd2caad13b21b3c811ce72312"}
Sep 16 17:45:04.887644 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:04.887064    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8" event={"ID":"a5bda94e-3f98-4e8d-a96b-326290b6d381","Type":"ContainerStarted","Data":"358fa3924ba216b303363bedceb73e272047f9af707bbd156b1d0ac51bb1cfd3"}
Sep 16 17:45:04.908162 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:04.908106    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8" podStartSLOduration=1.908086146 podStartE2EDuration="1.908086146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:45:04.907305713 +0000 UTC m=+1226.269984521" watchObservedRunningTime="2026-09-16 17:45:04.908086146 +0000 UTC m=+1226.270764936"
Sep 16 17:45:05.249990 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:05.249951    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:45:05.678514 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:05.678468    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8"
Sep 16 17:45:05.784341 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:05.784300    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:45:05.885964 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:05.885931    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:45:05.994098 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:05.994014    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:45:07.399315 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:07.399282    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8"]
Sep 16 17:45:07.421490 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:07.421452    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:45:07.915636 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:45:07.915602    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 17:45:07.915636 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:45:07.915636 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 17:45:07.915636 ip-10-0-136-122 kubenswrapper[2577]:  > containerID="449d51b6d47cc03c4e9b6a14331eadce7f54582cd2caad13b21b3c811ce72312" cmd=["/bin/sleep","15"]
Sep 16 17:45:07.915911 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:07.915650    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8" message="" err=<
Sep 16 17:45:07.915911 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:45:07.915911 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 17:45:07.915911 ip-10-0-136-122 kubenswrapper[2577]:  >
Sep 16 17:45:07.915911 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:45:07.915665    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 17:45:07.915911 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:45:07.915911 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 17:45:07.915911 ip-10-0-136-122 kubenswrapper[2577]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8" podUID="a5bda94e-3f98-4e8d-a96b-326290b6d381" containerName="main" containerID="cri-o://449d51b6d47cc03c4e9b6a14331eadce7f54582cd2caad13b21b3c811ce72312"
Sep 16 17:45:07.915911 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:07.915714    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8" podUID="a5bda94e-3f98-4e8d-a96b-326290b6d381" containerName="main" containerID="cri-o://449d51b6d47cc03c4e9b6a14331eadce7f54582cd2caad13b21b3c811ce72312" gracePeriod=60
Sep 16 17:45:08.180819 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:08.180794    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8"
Sep 16 17:45:08.275106 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:08.275080    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5bda94e-3f98-4e8d-a96b-326290b6d381-model-cache\") pod \"a5bda94e-3f98-4e8d-a96b-326290b6d381\" (UID: \"a5bda94e-3f98-4e8d-a96b-326290b6d381\") "
Sep 16 17:45:08.275287 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:08.275118    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5bda94e-3f98-4e8d-a96b-326290b6d381-tmp-dir\") pod \"a5bda94e-3f98-4e8d-a96b-326290b6d381\" (UID: \"a5bda94e-3f98-4e8d-a96b-326290b6d381\") "
Sep 16 17:45:08.275287 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:08.275175    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6z9n8\" (UniqueName: \"kubernetes.io/projected/a5bda94e-3f98-4e8d-a96b-326290b6d381-kube-api-access-6z9n8\") pod \"a5bda94e-3f98-4e8d-a96b-326290b6d381\" (UID: \"a5bda94e-3f98-4e8d-a96b-326290b6d381\") "
Sep 16 17:45:08.275287 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:08.275200    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5bda94e-3f98-4e8d-a96b-326290b6d381-tls-certs\") pod \"a5bda94e-3f98-4e8d-a96b-326290b6d381\" (UID: \"a5bda94e-3f98-4e8d-a96b-326290b6d381\") "
Sep 16 17:45:08.275287 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:08.275219    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5bda94e-3f98-4e8d-a96b-326290b6d381-dshm\") pod \"a5bda94e-3f98-4e8d-a96b-326290b6d381\" (UID: \"a5bda94e-3f98-4e8d-a96b-326290b6d381\") "
Sep 16 17:45:08.275509 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:08.275315    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5bda94e-3f98-4e8d-a96b-326290b6d381-home\") pod \"a5bda94e-3f98-4e8d-a96b-326290b6d381\" (UID: \"a5bda94e-3f98-4e8d-a96b-326290b6d381\") "
Sep 16 17:45:08.275509 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:08.275395    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5bda94e-3f98-4e8d-a96b-326290b6d381-model-cache" (OuterVolumeSpecName: "model-cache") pod "a5bda94e-3f98-4e8d-a96b-326290b6d381" (UID: "a5bda94e-3f98-4e8d-a96b-326290b6d381"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:45:08.275509 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:08.275470    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5bda94e-3f98-4e8d-a96b-326290b6d381-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a5bda94e-3f98-4e8d-a96b-326290b6d381" (UID: "a5bda94e-3f98-4e8d-a96b-326290b6d381"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:45:08.275658 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:08.275592    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5bda94e-3f98-4e8d-a96b-326290b6d381-home" (OuterVolumeSpecName: "home") pod "a5bda94e-3f98-4e8d-a96b-326290b6d381" (UID: "a5bda94e-3f98-4e8d-a96b-326290b6d381"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:45:08.275658 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:08.275648    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5bda94e-3f98-4e8d-a96b-326290b6d381-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:45:08.275748 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:08.275665    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5bda94e-3f98-4e8d-a96b-326290b6d381-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:45:08.275748 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:08.275680    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5bda94e-3f98-4e8d-a96b-326290b6d381-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:45:08.277692 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:08.277655    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5bda94e-3f98-4e8d-a96b-326290b6d381-dshm" (OuterVolumeSpecName: "dshm") pod "a5bda94e-3f98-4e8d-a96b-326290b6d381" (UID: "a5bda94e-3f98-4e8d-a96b-326290b6d381"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:45:08.277890 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:08.277850    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5bda94e-3f98-4e8d-a96b-326290b6d381-kube-api-access-6z9n8" (OuterVolumeSpecName: "kube-api-access-6z9n8") pod "a5bda94e-3f98-4e8d-a96b-326290b6d381" (UID: "a5bda94e-3f98-4e8d-a96b-326290b6d381"). InnerVolumeSpecName "kube-api-access-6z9n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:45:08.277890 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:08.277877    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5bda94e-3f98-4e8d-a96b-326290b6d381-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5bda94e-3f98-4e8d-a96b-326290b6d381" (UID: "a5bda94e-3f98-4e8d-a96b-326290b6d381"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:45:08.377074 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:08.377035    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z9n8\" (UniqueName: \"kubernetes.io/projected/a5bda94e-3f98-4e8d-a96b-326290b6d381-kube-api-access-6z9n8\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:45:08.377074 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:08.377066    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5bda94e-3f98-4e8d-a96b-326290b6d381-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:45:08.377074 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:08.377077    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5bda94e-3f98-4e8d-a96b-326290b6d381-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:45:08.912385 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:08.912352    2577 generic.go:356] "Generic (PLEG): container finished" podID="a5bda94e-3f98-4e8d-a96b-326290b6d381" containerID="449d51b6d47cc03c4e9b6a14331eadce7f54582cd2caad13b21b3c811ce72312" exitCode=0
Sep 16 17:45:08.912854 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:08.912399    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8"
Sep 16 17:45:08.912854 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:08.912448    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8" event={"ID":"a5bda94e-3f98-4e8d-a96b-326290b6d381","Type":"ContainerDied","Data":"449d51b6d47cc03c4e9b6a14331eadce7f54582cd2caad13b21b3c811ce72312"}
Sep 16 17:45:08.912854 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:08.912515    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8" event={"ID":"a5bda94e-3f98-4e8d-a96b-326290b6d381","Type":"ContainerDied","Data":"358fa3924ba216b303363bedceb73e272047f9af707bbd156b1d0ac51bb1cfd3"}
Sep 16 17:45:08.912854 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:08.912563    2577 scope.go:117] "RemoveContainer" containerID="449d51b6d47cc03c4e9b6a14331eadce7f54582cd2caad13b21b3c811ce72312"
Sep 16 17:45:08.922908 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:08.922659    2577 scope.go:117] "RemoveContainer" containerID="449d51b6d47cc03c4e9b6a14331eadce7f54582cd2caad13b21b3c811ce72312"
Sep 16 17:45:08.922987 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:45:08.922932    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"449d51b6d47cc03c4e9b6a14331eadce7f54582cd2caad13b21b3c811ce72312\": container with ID starting with 449d51b6d47cc03c4e9b6a14331eadce7f54582cd2caad13b21b3c811ce72312 not found: ID does not exist" containerID="449d51b6d47cc03c4e9b6a14331eadce7f54582cd2caad13b21b3c811ce72312"
Sep 16 17:45:08.922987 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:08.922957    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"449d51b6d47cc03c4e9b6a14331eadce7f54582cd2caad13b21b3c811ce72312"} err="failed to get container status \"449d51b6d47cc03c4e9b6a14331eadce7f54582cd2caad13b21b3c811ce72312\": rpc error: code = NotFound desc = could not find container \"449d51b6d47cc03c4e9b6a14331eadce7f54582cd2caad13b21b3c811ce72312\": container with ID starting with 449d51b6d47cc03c4e9b6a14331eadce7f54582cd2caad13b21b3c811ce72312 not found: ID does not exist"
Sep 16 17:45:08.938148 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:08.938113    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8"]
Sep 16 17:45:08.943538 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:08.943511    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvhrq8"]
Sep 16 17:45:09.081335 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:09.081302    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5bda94e-3f98-4e8d-a96b-326290b6d381" path="/var/lib/kubelet/pods/a5bda94e-3f98-4e8d-a96b-326290b6d381/volumes"
Sep 16 17:45:10.166341 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:10.166310    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:45:10.254699 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:10.254656    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:45:10.855058 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:10.855019    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:45:13.969072 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:13.968969    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="178b8d32-bf69-4588-9400-9afaa06ee302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 16 17:45:14.773710 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.773669    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"]
Sep 16 17:45:14.774108 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.774093    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5bda94e-3f98-4e8d-a96b-326290b6d381" containerName="main"
Sep 16 17:45:14.774161 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.774111    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5bda94e-3f98-4e8d-a96b-326290b6d381" containerName="main"
Sep 16 17:45:14.774212 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.774198    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5bda94e-3f98-4e8d-a96b-326290b6d381" containerName="main"
Sep 16 17:45:14.778302 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.778282    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"
Sep 16 17:45:14.782073 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.782048    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 16 17:45:14.782280 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.782261    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 16 17:45:14.782474 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.782459    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Sep 16 17:45:14.782574 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.782551    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 17:45:14.782947 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.782928    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-8x5nr\""
Sep 16 17:45:14.783538 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.783521    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 16 17:45:14.796777 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.796752    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"]
Sep 16 17:45:14.820398 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.820360    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:45:14.836421 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.836394    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-home\") pod \"auth-enabled-test-kserve-5dc549b8fc-d89z2\" (UID: \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"
Sep 16 17:45:14.836542 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.836424    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-model-cache\") pod \"auth-enabled-test-kserve-5dc549b8fc-d89z2\" (UID: \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"
Sep 16 17:45:14.836542 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.836462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-kserve-provision-location\") pod \"auth-enabled-test-kserve-5dc549b8fc-d89z2\" (UID: \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"
Sep 16 17:45:14.836542 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.836480    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-dshm\") pod \"auth-enabled-test-kserve-5dc549b8fc-d89z2\" (UID: \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"
Sep 16 17:45:14.836653 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.836525    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2228d\" (UniqueName: \"kubernetes.io/projected/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-kube-api-access-2228d\") pod \"auth-enabled-test-kserve-5dc549b8fc-d89z2\" (UID: \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"
Sep 16 17:45:14.836653 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.836584    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-cabundle-cert\") pod \"auth-enabled-test-kserve-5dc549b8fc-d89z2\" (UID: \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"
Sep 16 17:45:14.836653 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.836618    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-tmp-dir\") pod \"auth-enabled-test-kserve-5dc549b8fc-d89z2\" (UID: \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"
Sep 16 17:45:14.836748 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.836718    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-tls-certs\") pod \"auth-enabled-test-kserve-5dc549b8fc-d89z2\" (UID: \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"
Sep 16 17:45:14.938068 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.938029    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-cabundle-cert\") pod \"auth-enabled-test-kserve-5dc549b8fc-d89z2\" (UID: \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"
Sep 16 17:45:14.938068 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.938072    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-tmp-dir\") pod \"auth-enabled-test-kserve-5dc549b8fc-d89z2\" (UID: \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"
Sep 16 17:45:14.938312 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.938130    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-tls-certs\") pod \"auth-enabled-test-kserve-5dc549b8fc-d89z2\" (UID: \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"
Sep 16 17:45:14.938312 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.938173    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-home\") pod \"auth-enabled-test-kserve-5dc549b8fc-d89z2\" (UID: \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"
Sep 16 17:45:14.938312 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.938195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-model-cache\") pod \"auth-enabled-test-kserve-5dc549b8fc-d89z2\" (UID: \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"
Sep 16 17:45:14.938312 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.938248    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-kserve-provision-location\") pod \"auth-enabled-test-kserve-5dc549b8fc-d89z2\" (UID: \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"
Sep 16 17:45:14.938312 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.938288    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-dshm\") pod \"auth-enabled-test-kserve-5dc549b8fc-d89z2\" (UID: \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"
Sep 16 17:45:14.938555 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.938320    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2228d\" (UniqueName: \"kubernetes.io/projected/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-kube-api-access-2228d\") pod \"auth-enabled-test-kserve-5dc549b8fc-d89z2\" (UID: \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"
Sep 16 17:45:14.938698 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.938661    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-model-cache\") pod \"auth-enabled-test-kserve-5dc549b8fc-d89z2\" (UID: \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"
Sep 16 17:45:14.938832 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.938723    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-home\") pod \"auth-enabled-test-kserve-5dc549b8fc-d89z2\" (UID: \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"
Sep 16 17:45:14.938832 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.938780    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-tmp-dir\") pod \"auth-enabled-test-kserve-5dc549b8fc-d89z2\" (UID: \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"
Sep 16 17:45:14.938832 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.938820    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-kserve-provision-location\") pod \"auth-enabled-test-kserve-5dc549b8fc-d89z2\" (UID: \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"
Sep 16 17:45:14.939020 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.938891    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-cabundle-cert\") pod \"auth-enabled-test-kserve-5dc549b8fc-d89z2\" (UID: \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"
Sep 16 17:45:14.940664 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.940641    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-dshm\") pod \"auth-enabled-test-kserve-5dc549b8fc-d89z2\" (UID: \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"
Sep 16 17:45:14.941080 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.941064    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-tls-certs\") pod \"auth-enabled-test-kserve-5dc549b8fc-d89z2\" (UID: \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"
Sep 16 17:45:14.948207 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:14.948169    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2228d\" (UniqueName: \"kubernetes.io/projected/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-kube-api-access-2228d\") pod \"auth-enabled-test-kserve-5dc549b8fc-d89z2\" (UID: \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"
Sep 16 17:45:15.089586 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:15.089545    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"
Sep 16 17:45:15.340092 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:15.339997    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:45:15.370043 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:15.370007    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"]
Sep 16 17:45:15.372519 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:45:15.372490    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd259e77_1b4e_41da_9a7a_ac29eebc99b6.slice/crio-b2634caea3780c21112f6697e829214bc8a67e3bd614907b3e178d5f50ae671c WatchSource:0}: Error finding container b2634caea3780c21112f6697e829214bc8a67e3bd614907b3e178d5f50ae671c: Status 404 returned error can't find the container with id b2634caea3780c21112f6697e829214bc8a67e3bd614907b3e178d5f50ae671c
Sep 16 17:45:15.959057 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:15.959025    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2" event={"ID":"dd259e77-1b4e-41da-9a7a-ac29eebc99b6","Type":"ContainerStarted","Data":"7e641f800c0269b6e76496ff50ae8fd8924a62b3d0efc9c65c7b9308f45226c7"}
Sep 16 17:45:15.959224 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:15.959092    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2" event={"ID":"dd259e77-1b4e-41da-9a7a-ac29eebc99b6","Type":"ContainerStarted","Data":"b2634caea3780c21112f6697e829214bc8a67e3bd614907b3e178d5f50ae671c"}
Sep 16 17:45:16.496436 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:16.496389    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:45:17.972554 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:17.972512    2577 generic.go:356] "Generic (PLEG): container finished" podID="dd259e77-1b4e-41da-9a7a-ac29eebc99b6" containerID="7e641f800c0269b6e76496ff50ae8fd8924a62b3d0efc9c65c7b9308f45226c7" exitCode=0
Sep 16 17:45:17.972949 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:17.972579    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2" event={"ID":"dd259e77-1b4e-41da-9a7a-ac29eebc99b6","Type":"ContainerDied","Data":"7e641f800c0269b6e76496ff50ae8fd8924a62b3d0efc9c65c7b9308f45226c7"}
Sep 16 17:45:18.751115 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:18.751082    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:45:18.980541 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:18.980509    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2" event={"ID":"dd259e77-1b4e-41da-9a7a-ac29eebc99b6","Type":"ContainerStarted","Data":"831bca0f07c9e856c0652e82f8a3c3c9579254349e421f956bb9661a44ba2a4c"}
Sep 16 17:45:19.012964 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:19.012819    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2" podStartSLOduration=2.656771215 podStartE2EDuration="5.012801766s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.356030551s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:45:19.009163654 +0000 UTC m=+1240.371842440" watchObservedRunningTime="2026-09-16 17:45:19.012801766 +0000 UTC m=+1240.375480552"
Sep 16 17:45:19.089938 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:19.089904    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"
Sep 16 17:45:20.442705 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:20.442664    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:45:20.526581 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:20.526515    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:45:22.909434 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:22.909398    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:45:23.968921 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:23.968874    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="178b8d32-bf69-4588-9400-9afaa06ee302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 16 17:45:24.658620 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:24.658570    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:45:24.749277 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:24.749241    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:45:25.090381 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:25.090345    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"
Sep 16 17:45:25.092310 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:25.092275    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2" podUID="dd259e77-1b4e-41da-9a7a-ac29eebc99b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 16 17:45:30.729781 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:30.729735    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:45:33.969331 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:33.969236    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="178b8d32-bf69-4588-9400-9afaa06ee302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 16 17:45:35.090622 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:35.090580    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2" podUID="dd259e77-1b4e-41da-9a7a-ac29eebc99b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 16 17:45:43.969713 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:43.969662    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="178b8d32-bf69-4588-9400-9afaa06ee302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 16 17:45:45.090881 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:45.090824    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2" podUID="dd259e77-1b4e-41da-9a7a-ac29eebc99b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 16 17:45:53.968827 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:53.968772    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="178b8d32-bf69-4588-9400-9afaa06ee302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 16 17:45:55.090815 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:45:55.090764    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2" podUID="dd259e77-1b4e-41da-9a7a-ac29eebc99b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 16 17:46:03.969019 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:03.968969    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="178b8d32-bf69-4588-9400-9afaa06ee302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 16 17:46:05.090030 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:05.089973    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2" podUID="dd259e77-1b4e-41da-9a7a-ac29eebc99b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 16 17:46:13.978831 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:13.978799    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 17:46:13.986280 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:13.986247    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 17:46:15.090979 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:15.090935    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2" podUID="dd259e77-1b4e-41da-9a7a-ac29eebc99b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 16 17:46:19.446974 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:19.446934    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:46:19.460424 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:19.460380    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2"]
Sep 16 17:46:19.475026 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:19.474998    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 17:46:19.536551 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:19.536513    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:46:25.090790 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:25.090751    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2" podUID="dd259e77-1b4e-41da-9a7a-ac29eebc99b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 16 17:46:25.497103 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:25.497017    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2" podUID="33b9487d-8964-4dc8-956b-2ec1c59a6d67" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:46:34.480554 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:34.480514    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2" podUID="33b9487d-8964-4dc8-956b-2ec1c59a6d67" containerName="main" containerID="cri-o://1644ce23ea9b04d74592e9befde9c87e973fe644930823d84d07aa8d214ff816" gracePeriod=45
Sep 16 17:46:34.498605 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:34.498568    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="178b8d32-bf69-4588-9400-9afaa06ee302" containerName="main" containerID="cri-o://a1513a649bae7ac42a588f1aa8d60b0986aa9779025c2fc6332d84ef65508706" gracePeriod=45
Sep 16 17:46:34.979931 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:34.979892    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="178b8d32-bf69-4588-9400-9afaa06ee302" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 16 17:46:35.090381 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:35.090345    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2" podUID="dd259e77-1b4e-41da-9a7a-ac29eebc99b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 16 17:46:35.454512 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:35.454480    2577 generic.go:356] "Generic (PLEG): container finished" podID="178b8d32-bf69-4588-9400-9afaa06ee302" containerID="a1513a649bae7ac42a588f1aa8d60b0986aa9779025c2fc6332d84ef65508706" exitCode=0
Sep 16 17:46:35.454686 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:35.454563    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"178b8d32-bf69-4588-9400-9afaa06ee302","Type":"ContainerDied","Data":"a1513a649bae7ac42a588f1aa8d60b0986aa9779025c2fc6332d84ef65508706"}
Sep 16 17:46:35.482940 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:35.482915    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 17:46:35.496829 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:35.496793    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2" podUID="33b9487d-8964-4dc8-956b-2ec1c59a6d67" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:46:35.587802 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:35.587773    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/178b8d32-bf69-4588-9400-9afaa06ee302-cabundle-cert\") pod \"178b8d32-bf69-4588-9400-9afaa06ee302\" (UID: \"178b8d32-bf69-4588-9400-9afaa06ee302\") "
Sep 16 17:46:35.588013 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:35.587814    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7l9j\" (UniqueName: \"kubernetes.io/projected/178b8d32-bf69-4588-9400-9afaa06ee302-kube-api-access-m7l9j\") pod \"178b8d32-bf69-4588-9400-9afaa06ee302\" (UID: \"178b8d32-bf69-4588-9400-9afaa06ee302\") "
Sep 16 17:46:35.588013 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:35.587836    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/178b8d32-bf69-4588-9400-9afaa06ee302-tls-certs\") pod \"178b8d32-bf69-4588-9400-9afaa06ee302\" (UID: \"178b8d32-bf69-4588-9400-9afaa06ee302\") "
Sep 16 17:46:35.588013 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:35.587878    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/178b8d32-bf69-4588-9400-9afaa06ee302-dshm\") pod \"178b8d32-bf69-4588-9400-9afaa06ee302\" (UID: \"178b8d32-bf69-4588-9400-9afaa06ee302\") "
Sep 16 17:46:35.588013 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:35.587931    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/178b8d32-bf69-4588-9400-9afaa06ee302-model-cache\") pod \"178b8d32-bf69-4588-9400-9afaa06ee302\" (UID: \"178b8d32-bf69-4588-9400-9afaa06ee302\") "
Sep 16 17:46:35.588013 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:35.587956    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/178b8d32-bf69-4588-9400-9afaa06ee302-home\") pod \"178b8d32-bf69-4588-9400-9afaa06ee302\" (UID: \"178b8d32-bf69-4588-9400-9afaa06ee302\") "
Sep 16 17:46:35.588279 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:35.588037    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/178b8d32-bf69-4588-9400-9afaa06ee302-tmp-dir\") pod \"178b8d32-bf69-4588-9400-9afaa06ee302\" (UID: \"178b8d32-bf69-4588-9400-9afaa06ee302\") "
Sep 16 17:46:35.588279 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:35.588239    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178b8d32-bf69-4588-9400-9afaa06ee302-model-cache" (OuterVolumeSpecName: "model-cache") pod "178b8d32-bf69-4588-9400-9afaa06ee302" (UID: "178b8d32-bf69-4588-9400-9afaa06ee302"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:46:35.588385 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:35.588306    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/178b8d32-bf69-4588-9400-9afaa06ee302-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "178b8d32-bf69-4588-9400-9afaa06ee302" (UID: "178b8d32-bf69-4588-9400-9afaa06ee302"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:46:35.588680 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:35.588454    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/178b8d32-bf69-4588-9400-9afaa06ee302-kserve-provision-location\") pod \"178b8d32-bf69-4588-9400-9afaa06ee302\" (UID: \"178b8d32-bf69-4588-9400-9afaa06ee302\") "
Sep 16 17:46:35.588997 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:35.588912    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178b8d32-bf69-4588-9400-9afaa06ee302-home" (OuterVolumeSpecName: "home") pod "178b8d32-bf69-4588-9400-9afaa06ee302" (UID: "178b8d32-bf69-4588-9400-9afaa06ee302"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:46:35.588997 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:35.588950    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/178b8d32-bf69-4588-9400-9afaa06ee302-cabundle-cert\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:46:35.588997 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:35.588975    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/178b8d32-bf69-4588-9400-9afaa06ee302-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:46:35.590526 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:35.590449    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178b8d32-bf69-4588-9400-9afaa06ee302-dshm" (OuterVolumeSpecName: "dshm") pod "178b8d32-bf69-4588-9400-9afaa06ee302" (UID: "178b8d32-bf69-4588-9400-9afaa06ee302"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:46:35.590639 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:35.590590    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/178b8d32-bf69-4588-9400-9afaa06ee302-kube-api-access-m7l9j" (OuterVolumeSpecName: "kube-api-access-m7l9j") pod "178b8d32-bf69-4588-9400-9afaa06ee302" (UID: "178b8d32-bf69-4588-9400-9afaa06ee302"). InnerVolumeSpecName "kube-api-access-m7l9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:46:35.590639 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:35.590598    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/178b8d32-bf69-4588-9400-9afaa06ee302-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "178b8d32-bf69-4588-9400-9afaa06ee302" (UID: "178b8d32-bf69-4588-9400-9afaa06ee302"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:46:35.606402 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:35.606370    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178b8d32-bf69-4588-9400-9afaa06ee302-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "178b8d32-bf69-4588-9400-9afaa06ee302" (UID: "178b8d32-bf69-4588-9400-9afaa06ee302"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:46:35.609404 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:35.609382    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178b8d32-bf69-4588-9400-9afaa06ee302-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "178b8d32-bf69-4588-9400-9afaa06ee302" (UID: "178b8d32-bf69-4588-9400-9afaa06ee302"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:46:35.690238 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:35.690199    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7l9j\" (UniqueName: \"kubernetes.io/projected/178b8d32-bf69-4588-9400-9afaa06ee302-kube-api-access-m7l9j\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:46:35.690238 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:35.690231    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/178b8d32-bf69-4588-9400-9afaa06ee302-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:46:35.690238 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:35.690243    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/178b8d32-bf69-4588-9400-9afaa06ee302-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:46:35.690472 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:35.690253    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/178b8d32-bf69-4588-9400-9afaa06ee302-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:46:35.690472 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:35.690264    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/178b8d32-bf69-4588-9400-9afaa06ee302-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:46:35.690472 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:35.690275    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/178b8d32-bf69-4588-9400-9afaa06ee302-kserve-provision-location\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:46:36.462381 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:36.462348    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 17:46:36.462591 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:36.462376    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"178b8d32-bf69-4588-9400-9afaa06ee302","Type":"ContainerDied","Data":"c2e00d705fc9213d58bbbfc2a8745592d6eac64c871b7ae1b93b301f896d67dc"}
Sep 16 17:46:36.462591 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:36.462445    2577 scope.go:117] "RemoveContainer" containerID="a1513a649bae7ac42a588f1aa8d60b0986aa9779025c2fc6332d84ef65508706"
Sep 16 17:46:36.471702 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:36.471683    2577 scope.go:117] "RemoveContainer" containerID="8137640ad928efa52635a47882e67a7d6b6fab895a6dd832db276bc402e01369"
Sep 16 17:46:36.490338 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:36.490306    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 17:46:36.492490 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:36.492462    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 17:46:37.080992 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:37.080954    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="178b8d32-bf69-4588-9400-9afaa06ee302" path="/var/lib/kubelet/pods/178b8d32-bf69-4588-9400-9afaa06ee302/volumes"
Sep 16 17:46:45.090824 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:45.090779    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2" podUID="dd259e77-1b4e-41da-9a7a-ac29eebc99b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 16 17:46:45.496913 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:45.496798    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2" podUID="33b9487d-8964-4dc8-956b-2ec1c59a6d67" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:46:45.497094 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:45.496925    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2"
Sep 16 17:46:55.089972 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:55.089931    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2" podUID="dd259e77-1b4e-41da-9a7a-ac29eebc99b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 16 17:46:55.496858 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:55.496764    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2" podUID="33b9487d-8964-4dc8-956b-2ec1c59a6d67" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:46:57.558249 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:46:57.558207    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:47:04.619911 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:04.619877    2577 generic.go:356] "Generic (PLEG): container finished" podID="33b9487d-8964-4dc8-956b-2ec1c59a6d67" containerID="1644ce23ea9b04d74592e9befde9c87e973fe644930823d84d07aa8d214ff816" exitCode=0
Sep 16 17:47:04.620375 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:04.619905    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2" event={"ID":"33b9487d-8964-4dc8-956b-2ec1c59a6d67","Type":"ContainerDied","Data":"1644ce23ea9b04d74592e9befde9c87e973fe644930823d84d07aa8d214ff816"}
Sep 16 17:47:04.747169 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:04.747099    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2"
Sep 16 17:47:04.756225 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:04.756202    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n46rv\" (UniqueName: \"kubernetes.io/projected/33b9487d-8964-4dc8-956b-2ec1c59a6d67-kube-api-access-n46rv\") pod \"33b9487d-8964-4dc8-956b-2ec1c59a6d67\" (UID: \"33b9487d-8964-4dc8-956b-2ec1c59a6d67\") "
Sep 16 17:47:04.756364 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:04.756292    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b9487d-8964-4dc8-956b-2ec1c59a6d67-tls-certs\") pod \"33b9487d-8964-4dc8-956b-2ec1c59a6d67\" (UID: \"33b9487d-8964-4dc8-956b-2ec1c59a6d67\") "
Sep 16 17:47:04.758346 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:04.758311    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b9487d-8964-4dc8-956b-2ec1c59a6d67-kube-api-access-n46rv" (OuterVolumeSpecName: "kube-api-access-n46rv") pod "33b9487d-8964-4dc8-956b-2ec1c59a6d67" (UID: "33b9487d-8964-4dc8-956b-2ec1c59a6d67"). InnerVolumeSpecName "kube-api-access-n46rv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:47:04.758453 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:04.758327    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33b9487d-8964-4dc8-956b-2ec1c59a6d67-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "33b9487d-8964-4dc8-956b-2ec1c59a6d67" (UID: "33b9487d-8964-4dc8-956b-2ec1c59a6d67"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:47:04.857694 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:04.857659    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n46rv\" (UniqueName: \"kubernetes.io/projected/33b9487d-8964-4dc8-956b-2ec1c59a6d67-kube-api-access-n46rv\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:47:04.857694 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:04.857688    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b9487d-8964-4dc8-956b-2ec1c59a6d67-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:47:05.090663 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:05.090627    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2" podUID="dd259e77-1b4e-41da-9a7a-ac29eebc99b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 16 17:47:05.626770 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:05.626729    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2" event={"ID":"33b9487d-8964-4dc8-956b-2ec1c59a6d67","Type":"ContainerDied","Data":"300a87b3bacd22baf5d7ae612868f3114dbe870c9884f83c53e66c0c6d521de1"}
Sep 16 17:47:05.627183 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:05.626788    2577 scope.go:117] "RemoveContainer" containerID="1644ce23ea9b04d74592e9befde9c87e973fe644930823d84d07aa8d214ff816"
Sep 16 17:47:05.627183 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:05.626742    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2"
Sep 16 17:47:05.645148 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:05.645125    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2"]
Sep 16 17:47:05.652143 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:05.652119    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schexmsp2"]
Sep 16 17:47:07.084266 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:07.084227    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33b9487d-8964-4dc8-956b-2ec1c59a6d67" path="/var/lib/kubelet/pods/33b9487d-8964-4dc8-956b-2ec1c59a6d67/volumes"
Sep 16 17:47:08.082663 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:08.082615    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:47:09.643265 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:09.643223    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:47:09.798818 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:09.798781    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:47:15.100929 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:15.100902    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"
Sep 16 17:47:15.108799 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:15.108772    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"
Sep 16 17:47:16.695567 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:16.695527    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:47:20.874927 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:20.874887    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6"]
Sep 16 17:47:20.875325 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:20.875309    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="178b8d32-bf69-4588-9400-9afaa06ee302" containerName="main"
Sep 16 17:47:20.875325 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:20.875323    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="178b8d32-bf69-4588-9400-9afaa06ee302" containerName="main"
Sep 16 17:47:20.875407 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:20.875330    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="178b8d32-bf69-4588-9400-9afaa06ee302" containerName="storage-initializer"
Sep 16 17:47:20.875407 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:20.875337    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="178b8d32-bf69-4588-9400-9afaa06ee302" containerName="storage-initializer"
Sep 16 17:47:20.875407 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:20.875350    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33b9487d-8964-4dc8-956b-2ec1c59a6d67" containerName="main"
Sep 16 17:47:20.875407 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:20.875358    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="33b9487d-8964-4dc8-956b-2ec1c59a6d67" containerName="main"
Sep 16 17:47:20.875549 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:20.875419    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="33b9487d-8964-4dc8-956b-2ec1c59a6d67" containerName="main"
Sep 16 17:47:20.875549 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:20.875426    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="178b8d32-bf69-4588-9400-9afaa06ee302" containerName="main"
Sep 16 17:47:20.878718 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:20.878689    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6"
Sep 16 17:47:20.881791 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:20.881767    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 16 17:47:20.882777 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:20.882760    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 16 17:47:20.882936 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:20.882914    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 16 17:47:20.883043 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:20.882996    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-pd9ch\""
Sep 16 17:47:20.889793 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:20.889772    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6"]
Sep 16 17:47:20.901602 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:20.901564    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:47:21.000408 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:21.000371    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30951b07-0f98-4daf-8c85-c935786707dc-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-g74v6\" (UID: \"30951b07-0f98-4daf-8c85-c935786707dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6"
Sep 16 17:47:21.000572 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:21.000448    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30951b07-0f98-4daf-8c85-c935786707dc-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-g74v6\" (UID: \"30951b07-0f98-4daf-8c85-c935786707dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6"
Sep 16 17:47:21.000572 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:21.000472    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30951b07-0f98-4daf-8c85-c935786707dc-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-g74v6\" (UID: \"30951b07-0f98-4daf-8c85-c935786707dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6"
Sep 16 17:47:21.000572 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:21.000487    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30951b07-0f98-4daf-8c85-c935786707dc-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-g74v6\" (UID: \"30951b07-0f98-4daf-8c85-c935786707dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6"
Sep 16 17:47:21.000572 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:21.000505    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h74xm\" (UniqueName: \"kubernetes.io/projected/30951b07-0f98-4daf-8c85-c935786707dc-kube-api-access-h74xm\") pod \"scheduler-inline-config-test-kserve-8655bc757-g74v6\" (UID: \"30951b07-0f98-4daf-8c85-c935786707dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6"
Sep 16 17:47:21.000710 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:21.000572    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30951b07-0f98-4daf-8c85-c935786707dc-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-g74v6\" (UID: \"30951b07-0f98-4daf-8c85-c935786707dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6"
Sep 16 17:47:21.101120 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:21.101087    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30951b07-0f98-4daf-8c85-c935786707dc-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-g74v6\" (UID: \"30951b07-0f98-4daf-8c85-c935786707dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6"
Sep 16 17:47:21.101312 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:21.101134    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30951b07-0f98-4daf-8c85-c935786707dc-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-g74v6\" (UID: \"30951b07-0f98-4daf-8c85-c935786707dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6"
Sep 16 17:47:21.101312 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:21.101156    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30951b07-0f98-4daf-8c85-c935786707dc-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-g74v6\" (UID: \"30951b07-0f98-4daf-8c85-c935786707dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6"
Sep 16 17:47:21.101312 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:21.101180    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h74xm\" (UniqueName: \"kubernetes.io/projected/30951b07-0f98-4daf-8c85-c935786707dc-kube-api-access-h74xm\") pod \"scheduler-inline-config-test-kserve-8655bc757-g74v6\" (UID: \"30951b07-0f98-4daf-8c85-c935786707dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6"
Sep 16 17:47:21.101312 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:21.101210    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30951b07-0f98-4daf-8c85-c935786707dc-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-g74v6\" (UID: \"30951b07-0f98-4daf-8c85-c935786707dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6"
Sep 16 17:47:21.101312 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:21.101287    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30951b07-0f98-4daf-8c85-c935786707dc-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-g74v6\" (UID: \"30951b07-0f98-4daf-8c85-c935786707dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6"
Sep 16 17:47:21.101679 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:21.101652    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30951b07-0f98-4daf-8c85-c935786707dc-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-g74v6\" (UID: \"30951b07-0f98-4daf-8c85-c935786707dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6"
Sep 16 17:47:21.101781 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:21.101691    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30951b07-0f98-4daf-8c85-c935786707dc-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-g74v6\" (UID: \"30951b07-0f98-4daf-8c85-c935786707dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6"
Sep 16 17:47:21.101830 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:21.101805    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30951b07-0f98-4daf-8c85-c935786707dc-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-g74v6\" (UID: \"30951b07-0f98-4daf-8c85-c935786707dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6"
Sep 16 17:47:21.103987 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:21.103965    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30951b07-0f98-4daf-8c85-c935786707dc-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-g74v6\" (UID: \"30951b07-0f98-4daf-8c85-c935786707dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6"
Sep 16 17:47:21.104140 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:21.104124    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30951b07-0f98-4daf-8c85-c935786707dc-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-g74v6\" (UID: \"30951b07-0f98-4daf-8c85-c935786707dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6"
Sep 16 17:47:21.110235 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:21.110214    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h74xm\" (UniqueName: \"kubernetes.io/projected/30951b07-0f98-4daf-8c85-c935786707dc-kube-api-access-h74xm\") pod \"scheduler-inline-config-test-kserve-8655bc757-g74v6\" (UID: \"30951b07-0f98-4daf-8c85-c935786707dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6"
Sep 16 17:47:21.191340 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:21.191246    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6"
Sep 16 17:47:21.202673 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:21.202637    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:47:21.475168 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:21.475134    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6"]
Sep 16 17:47:21.481857 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:47:21.481823    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30951b07_0f98_4daf_8c85_c935786707dc.slice/crio-a0975c0eb374093ae7c192f1bde44b5ff65c311d5f19fb5f5935cc322387a79e WatchSource:0}: Error finding container a0975c0eb374093ae7c192f1bde44b5ff65c311d5f19fb5f5935cc322387a79e: Status 404 returned error can't find the container with id a0975c0eb374093ae7c192f1bde44b5ff65c311d5f19fb5f5935cc322387a79e
Sep 16 17:47:21.716093 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:21.716056    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6" event={"ID":"30951b07-0f98-4daf-8c85-c935786707dc","Type":"ContainerStarted","Data":"a459c08832a6e315249455c276030c3f4740ab5b20b37004bf63666b63b963ed"}
Sep 16 17:47:21.716299 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:21.716108    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6" event={"ID":"30951b07-0f98-4daf-8c85-c935786707dc","Type":"ContainerStarted","Data":"a0975c0eb374093ae7c192f1bde44b5ff65c311d5f19fb5f5935cc322387a79e"}
Sep 16 17:47:21.747549 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:21.747442    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6" podStartSLOduration=1.74742132 podStartE2EDuration="1.74742132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:47:21.746463938 +0000 UTC m=+1363.109142735" watchObservedRunningTime="2026-09-16 17:47:21.74742132 +0000 UTC m=+1363.110100107"
Sep 16 17:47:22.191459 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:22.191425    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6"
Sep 16 17:47:24.048338 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:24.048296    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:47:24.139501 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:24.139460    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:47:31.191463 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:31.191422    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6"
Sep 16 17:47:31.204104 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:31.204075    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6"
Sep 16 17:47:31.785645 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:31.785610    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6"
Sep 16 17:47:41.075628 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:41.075592    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:47:42.621618 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:42.621583    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:47:42.623096 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:42.623067    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"]
Sep 16 17:47:42.710144 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:42.710108    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:47:55.629523 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:55.629488    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6"]
Sep 16 17:47:55.638593 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:47:55.638560    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 17:47:55.638593 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:47:55.638593 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 17:47:55.638593 ip-10-0-136-122 kubenswrapper[2577]:  > containerID="a459c08832a6e315249455c276030c3f4740ab5b20b37004bf63666b63b963ed" cmd=["/bin/sleep","15"]
Sep 16 17:47:55.638913 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:55.638618    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6" message="" err=<
Sep 16 17:47:55.638913 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:47:55.638913 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 17:47:55.638913 ip-10-0-136-122 kubenswrapper[2577]:  >
Sep 16 17:47:55.638913 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:47:55.638636    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 17:47:55.638913 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:47:55.638913 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 17:47:55.638913 ip-10-0-136-122 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6" podUID="30951b07-0f98-4daf-8c85-c935786707dc" containerName="main" containerID="cri-o://a459c08832a6e315249455c276030c3f4740ab5b20b37004bf63666b63b963ed"
Sep 16 17:47:55.638913 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:55.638678    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6" podUID="30951b07-0f98-4daf-8c85-c935786707dc" containerName="main" containerID="cri-o://a459c08832a6e315249455c276030c3f4740ab5b20b37004bf63666b63b963ed" gracePeriod=60
Sep 16 17:47:55.649472 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:55.649445    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:47:55.747530 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:55.747493    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:47:55.772307 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:55.772276    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6" podUID="30951b07-0f98-4daf-8c85-c935786707dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 16 17:47:55.900286 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:55.900198    2577 generic.go:356] "Generic (PLEG): container finished" podID="30951b07-0f98-4daf-8c85-c935786707dc" containerID="a459c08832a6e315249455c276030c3f4740ab5b20b37004bf63666b63b963ed" exitCode=0
Sep 16 17:47:55.900457 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:55.900281    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6" event={"ID":"30951b07-0f98-4daf-8c85-c935786707dc","Type":"ContainerDied","Data":"a459c08832a6e315249455c276030c3f4740ab5b20b37004bf63666b63b963ed"}
Sep 16 17:47:55.986621 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:55.986597    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6"
Sep 16 17:47:55.998014 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:55.997992    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30951b07-0f98-4daf-8c85-c935786707dc-tmp-dir\") pod \"30951b07-0f98-4daf-8c85-c935786707dc\" (UID: \"30951b07-0f98-4daf-8c85-c935786707dc\") "
Sep 16 17:47:55.998121 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:55.998029    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h74xm\" (UniqueName: \"kubernetes.io/projected/30951b07-0f98-4daf-8c85-c935786707dc-kube-api-access-h74xm\") pod \"30951b07-0f98-4daf-8c85-c935786707dc\" (UID: \"30951b07-0f98-4daf-8c85-c935786707dc\") "
Sep 16 17:47:55.998121 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:55.998067    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30951b07-0f98-4daf-8c85-c935786707dc-tls-certs\") pod \"30951b07-0f98-4daf-8c85-c935786707dc\" (UID: \"30951b07-0f98-4daf-8c85-c935786707dc\") "
Sep 16 17:47:55.998121 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:55.998093    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30951b07-0f98-4daf-8c85-c935786707dc-home\") pod \"30951b07-0f98-4daf-8c85-c935786707dc\" (UID: \"30951b07-0f98-4daf-8c85-c935786707dc\") "
Sep 16 17:47:55.998249 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:55.998176    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30951b07-0f98-4daf-8c85-c935786707dc-dshm\") pod \"30951b07-0f98-4daf-8c85-c935786707dc\" (UID: \"30951b07-0f98-4daf-8c85-c935786707dc\") "
Sep 16 17:47:55.998249 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:55.998209    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30951b07-0f98-4daf-8c85-c935786707dc-model-cache\") pod \"30951b07-0f98-4daf-8c85-c935786707dc\" (UID: \"30951b07-0f98-4daf-8c85-c935786707dc\") "
Sep 16 17:47:55.998351 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:55.998263    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30951b07-0f98-4daf-8c85-c935786707dc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "30951b07-0f98-4daf-8c85-c935786707dc" (UID: "30951b07-0f98-4daf-8c85-c935786707dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:47:55.998457 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:55.998440    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30951b07-0f98-4daf-8c85-c935786707dc-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:47:55.998518 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:55.998463    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30951b07-0f98-4daf-8c85-c935786707dc-home" (OuterVolumeSpecName: "home") pod "30951b07-0f98-4daf-8c85-c935786707dc" (UID: "30951b07-0f98-4daf-8c85-c935786707dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:47:55.998518 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:55.998496    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30951b07-0f98-4daf-8c85-c935786707dc-model-cache" (OuterVolumeSpecName: "model-cache") pod "30951b07-0f98-4daf-8c85-c935786707dc" (UID: "30951b07-0f98-4daf-8c85-c935786707dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:47:56.000301 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:56.000268    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30951b07-0f98-4daf-8c85-c935786707dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "30951b07-0f98-4daf-8c85-c935786707dc" (UID: "30951b07-0f98-4daf-8c85-c935786707dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:47:56.000301 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:56.000276    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30951b07-0f98-4daf-8c85-c935786707dc-dshm" (OuterVolumeSpecName: "dshm") pod "30951b07-0f98-4daf-8c85-c935786707dc" (UID: "30951b07-0f98-4daf-8c85-c935786707dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:47:56.000415 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:56.000341    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30951b07-0f98-4daf-8c85-c935786707dc-kube-api-access-h74xm" (OuterVolumeSpecName: "kube-api-access-h74xm") pod "30951b07-0f98-4daf-8c85-c935786707dc" (UID: "30951b07-0f98-4daf-8c85-c935786707dc"). InnerVolumeSpecName "kube-api-access-h74xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:47:56.099031 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:56.099002    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30951b07-0f98-4daf-8c85-c935786707dc-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:47:56.099031 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:56.099027    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30951b07-0f98-4daf-8c85-c935786707dc-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:47:56.099227 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:56.099044    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h74xm\" (UniqueName: \"kubernetes.io/projected/30951b07-0f98-4daf-8c85-c935786707dc-kube-api-access-h74xm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:47:56.099227 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:56.099054    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30951b07-0f98-4daf-8c85-c935786707dc-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:47:56.099227 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:56.099063    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30951b07-0f98-4daf-8c85-c935786707dc-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:47:56.907233 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:56.907203    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6"
Sep 16 17:47:56.907694 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:56.907239    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6" event={"ID":"30951b07-0f98-4daf-8c85-c935786707dc","Type":"ContainerDied","Data":"a0975c0eb374093ae7c192f1bde44b5ff65c311d5f19fb5f5935cc322387a79e"}
Sep 16 17:47:56.907694 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:56.907311    2577 scope.go:117] "RemoveContainer" containerID="a459c08832a6e315249455c276030c3f4740ab5b20b37004bf63666b63b963ed"
Sep 16 17:47:56.935401 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:56.935363    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6"]
Sep 16 17:47:56.935545 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:56.935421    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-g74v6"]
Sep 16 17:47:57.080744 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:57.080710    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30951b07-0f98-4daf-8c85-c935786707dc" path="/var/lib/kubelet/pods/30951b07-0f98-4daf-8c85-c935786707dc/volumes"
Sep 16 17:47:57.645703 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:57.645655    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2" podUID="dd259e77-1b4e-41da-9a7a-ac29eebc99b6" containerName="main" containerID="cri-o://831bca0f07c9e856c0652e82f8a3c3c9579254349e421f956bb9661a44ba2a4c" gracePeriod=45
Sep 16 17:47:58.102090 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.102046    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2" podUID="dd259e77-1b4e-41da-9a7a-ac29eebc99b6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 16 17:47:58.316916 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.316856    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"
Sep 16 17:47:58.416337 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.416250    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-model-cache\") pod \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\" (UID: \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\") "
Sep 16 17:47:58.416337 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.416298    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-tls-certs\") pod \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\" (UID: \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\") "
Sep 16 17:47:58.416337 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.416325    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-home\") pod \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\" (UID: \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\") "
Sep 16 17:47:58.416616 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.416361    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-kserve-provision-location\") pod \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\" (UID: \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\") "
Sep 16 17:47:58.416616 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.416375    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-dshm\") pod \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\" (UID: \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\") "
Sep 16 17:47:58.416616 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.416393    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2228d\" (UniqueName: \"kubernetes.io/projected/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-kube-api-access-2228d\") pod \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\" (UID: \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\") "
Sep 16 17:47:58.416616 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.416411    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-cabundle-cert\") pod \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\" (UID: \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\") "
Sep 16 17:47:58.416616 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.416495    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-tmp-dir\") pod \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\" (UID: \"dd259e77-1b4e-41da-9a7a-ac29eebc99b6\") "
Sep 16 17:47:58.416616 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.416565    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-model-cache" (OuterVolumeSpecName: "model-cache") pod "dd259e77-1b4e-41da-9a7a-ac29eebc99b6" (UID: "dd259e77-1b4e-41da-9a7a-ac29eebc99b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:47:58.416952 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.416925    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dd259e77-1b4e-41da-9a7a-ac29eebc99b6" (UID: "dd259e77-1b4e-41da-9a7a-ac29eebc99b6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:47:58.417142 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.417123    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:47:58.417226 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.417148    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-cabundle-cert\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:47:58.417226 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.417128    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-home" (OuterVolumeSpecName: "home") pod "dd259e77-1b4e-41da-9a7a-ac29eebc99b6" (UID: "dd259e77-1b4e-41da-9a7a-ac29eebc99b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:47:58.418901 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.418846    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-dshm" (OuterVolumeSpecName: "dshm") pod "dd259e77-1b4e-41da-9a7a-ac29eebc99b6" (UID: "dd259e77-1b4e-41da-9a7a-ac29eebc99b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:47:58.419119 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.419080    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dd259e77-1b4e-41da-9a7a-ac29eebc99b6" (UID: "dd259e77-1b4e-41da-9a7a-ac29eebc99b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:47:58.419119 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.419089    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-kube-api-access-2228d" (OuterVolumeSpecName: "kube-api-access-2228d") pod "dd259e77-1b4e-41da-9a7a-ac29eebc99b6" (UID: "dd259e77-1b4e-41da-9a7a-ac29eebc99b6"). InnerVolumeSpecName "kube-api-access-2228d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:47:58.437586 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.437563    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dd259e77-1b4e-41da-9a7a-ac29eebc99b6" (UID: "dd259e77-1b4e-41da-9a7a-ac29eebc99b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:47:58.437771 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.437748    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dd259e77-1b4e-41da-9a7a-ac29eebc99b6" (UID: "dd259e77-1b4e-41da-9a7a-ac29eebc99b6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:47:58.517565 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.517537    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-kserve-provision-location\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:47:58.517565 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.517563    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:47:58.517727 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.517574    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2228d\" (UniqueName: \"kubernetes.io/projected/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-kube-api-access-2228d\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:47:58.517727 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.517585    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:47:58.517727 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.517594    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:47:58.517727 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.517601    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd259e77-1b4e-41da-9a7a-ac29eebc99b6-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:47:58.922176 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.922144    2577 generic.go:356] "Generic (PLEG): container finished" podID="dd259e77-1b4e-41da-9a7a-ac29eebc99b6" containerID="831bca0f07c9e856c0652e82f8a3c3c9579254349e421f956bb9661a44ba2a4c" exitCode=0
Sep 16 17:47:58.922342 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.922192    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"
Sep 16 17:47:58.922342 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.922225    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2" event={"ID":"dd259e77-1b4e-41da-9a7a-ac29eebc99b6","Type":"ContainerDied","Data":"831bca0f07c9e856c0652e82f8a3c3c9579254349e421f956bb9661a44ba2a4c"}
Sep 16 17:47:58.922342 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.922278    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2" event={"ID":"dd259e77-1b4e-41da-9a7a-ac29eebc99b6","Type":"ContainerDied","Data":"b2634caea3780c21112f6697e829214bc8a67e3bd614907b3e178d5f50ae671c"}
Sep 16 17:47:58.922342 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.922303    2577 scope.go:117] "RemoveContainer" containerID="831bca0f07c9e856c0652e82f8a3c3c9579254349e421f956bb9661a44ba2a4c"
Sep 16 17:47:58.931638 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.931620    2577 scope.go:117] "RemoveContainer" containerID="7e641f800c0269b6e76496ff50ae8fd8924a62b3d0efc9c65c7b9308f45226c7"
Sep 16 17:47:58.939307 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.939291    2577 scope.go:117] "RemoveContainer" containerID="831bca0f07c9e856c0652e82f8a3c3c9579254349e421f956bb9661a44ba2a4c"
Sep 16 17:47:58.939560 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:47:58.939541    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"831bca0f07c9e856c0652e82f8a3c3c9579254349e421f956bb9661a44ba2a4c\": container with ID starting with 831bca0f07c9e856c0652e82f8a3c3c9579254349e421f956bb9661a44ba2a4c not found: ID does not exist" containerID="831bca0f07c9e856c0652e82f8a3c3c9579254349e421f956bb9661a44ba2a4c"
Sep 16 17:47:58.939623 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.939566    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"831bca0f07c9e856c0652e82f8a3c3c9579254349e421f956bb9661a44ba2a4c"} err="failed to get container status \"831bca0f07c9e856c0652e82f8a3c3c9579254349e421f956bb9661a44ba2a4c\": rpc error: code = NotFound desc = could not find container \"831bca0f07c9e856c0652e82f8a3c3c9579254349e421f956bb9661a44ba2a4c\": container with ID starting with 831bca0f07c9e856c0652e82f8a3c3c9579254349e421f956bb9661a44ba2a4c not found: ID does not exist"
Sep 16 17:47:58.939623 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.939582    2577 scope.go:117] "RemoveContainer" containerID="7e641f800c0269b6e76496ff50ae8fd8924a62b3d0efc9c65c7b9308f45226c7"
Sep 16 17:47:58.939821 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:47:58.939805    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e641f800c0269b6e76496ff50ae8fd8924a62b3d0efc9c65c7b9308f45226c7\": container with ID starting with 7e641f800c0269b6e76496ff50ae8fd8924a62b3d0efc9c65c7b9308f45226c7 not found: ID does not exist" containerID="7e641f800c0269b6e76496ff50ae8fd8924a62b3d0efc9c65c7b9308f45226c7"
Sep 16 17:47:58.939875 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.939827    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e641f800c0269b6e76496ff50ae8fd8924a62b3d0efc9c65c7b9308f45226c7"} err="failed to get container status \"7e641f800c0269b6e76496ff50ae8fd8924a62b3d0efc9c65c7b9308f45226c7\": rpc error: code = NotFound desc = could not find container \"7e641f800c0269b6e76496ff50ae8fd8924a62b3d0efc9c65c7b9308f45226c7\": container with ID starting with 7e641f800c0269b6e76496ff50ae8fd8924a62b3d0efc9c65c7b9308f45226c7 not found: ID does not exist"
Sep 16 17:47:58.959899 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.959856    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"]
Sep 16 17:47:58.968125 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:58.968105    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5dc549b8fc-d89z2"]
Sep 16 17:47:59.081833 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:47:59.081805    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd259e77-1b4e-41da-9a7a-ac29eebc99b6" path="/var/lib/kubelet/pods/dd259e77-1b4e-41da-9a7a-ac29eebc99b6/volumes"
Sep 16 17:48:29.054065 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:29.054020    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:48:30.856496 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:30.856458    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:48:35.364619 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.364584    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"]
Sep 16 17:48:35.365096 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.364942    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30951b07-0f98-4daf-8c85-c935786707dc" containerName="main"
Sep 16 17:48:35.365096 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.364955    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="30951b07-0f98-4daf-8c85-c935786707dc" containerName="main"
Sep 16 17:48:35.365096 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.364979    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd259e77-1b4e-41da-9a7a-ac29eebc99b6" containerName="storage-initializer"
Sep 16 17:48:35.365096 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.364985    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd259e77-1b4e-41da-9a7a-ac29eebc99b6" containerName="storage-initializer"
Sep 16 17:48:35.365096 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.364993    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd259e77-1b4e-41da-9a7a-ac29eebc99b6" containerName="main"
Sep 16 17:48:35.365096 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.364998    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd259e77-1b4e-41da-9a7a-ac29eebc99b6" containerName="main"
Sep 16 17:48:35.365096 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.365049    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="30951b07-0f98-4daf-8c85-c935786707dc" containerName="main"
Sep 16 17:48:35.365096 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.365056    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd259e77-1b4e-41da-9a7a-ac29eebc99b6" containerName="main"
Sep 16 17:48:35.368298 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.368280    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"
Sep 16 17:48:35.371093 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.371068    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 16 17:48:35.371093 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.371072    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Sep 16 17:48:35.371399 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.371382    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-2c8gk\""
Sep 16 17:48:35.371490 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.371445    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 16 17:48:35.371490 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.371471    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 17:48:35.371490 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.371480    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 16 17:48:35.383210 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.383188    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"]
Sep 16 17:48:35.397873 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.397823    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:48:35.502113 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.502074    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d23aec2-d092-4772-bd17-d7dff65c53fe-home\") pod \"auth-invalid-token-test-kserve-5fc974f796-zfp58\" (UID: \"3d23aec2-d092-4772-bd17-d7dff65c53fe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"
Sep 16 17:48:35.502113 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.502116    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d23aec2-d092-4772-bd17-d7dff65c53fe-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5fc974f796-zfp58\" (UID: \"3d23aec2-d092-4772-bd17-d7dff65c53fe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"
Sep 16 17:48:35.502331 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.502144    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d23aec2-d092-4772-bd17-d7dff65c53fe-model-cache\") pod \"auth-invalid-token-test-kserve-5fc974f796-zfp58\" (UID: \"3d23aec2-d092-4772-bd17-d7dff65c53fe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"
Sep 16 17:48:35.502331 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.502201    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d23aec2-d092-4772-bd17-d7dff65c53fe-dshm\") pod \"auth-invalid-token-test-kserve-5fc974f796-zfp58\" (UID: \"3d23aec2-d092-4772-bd17-d7dff65c53fe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"
Sep 16 17:48:35.502331 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.502228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d23aec2-d092-4772-bd17-d7dff65c53fe-cabundle-cert\") pod \"auth-invalid-token-test-kserve-5fc974f796-zfp58\" (UID: \"3d23aec2-d092-4772-bd17-d7dff65c53fe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"
Sep 16 17:48:35.502331 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.502248    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d23aec2-d092-4772-bd17-d7dff65c53fe-tls-certs\") pod \"auth-invalid-token-test-kserve-5fc974f796-zfp58\" (UID: \"3d23aec2-d092-4772-bd17-d7dff65c53fe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"
Sep 16 17:48:35.502331 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.502282    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5grhz\" (UniqueName: \"kubernetes.io/projected/3d23aec2-d092-4772-bd17-d7dff65c53fe-kube-api-access-5grhz\") pod \"auth-invalid-token-test-kserve-5fc974f796-zfp58\" (UID: \"3d23aec2-d092-4772-bd17-d7dff65c53fe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"
Sep 16 17:48:35.502521 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.502333    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d23aec2-d092-4772-bd17-d7dff65c53fe-tmp-dir\") pod \"auth-invalid-token-test-kserve-5fc974f796-zfp58\" (UID: \"3d23aec2-d092-4772-bd17-d7dff65c53fe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"
Sep 16 17:48:35.603215 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.603180    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d23aec2-d092-4772-bd17-d7dff65c53fe-model-cache\") pod \"auth-invalid-token-test-kserve-5fc974f796-zfp58\" (UID: \"3d23aec2-d092-4772-bd17-d7dff65c53fe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"
Sep 16 17:48:35.603215 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.603220    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d23aec2-d092-4772-bd17-d7dff65c53fe-dshm\") pod \"auth-invalid-token-test-kserve-5fc974f796-zfp58\" (UID: \"3d23aec2-d092-4772-bd17-d7dff65c53fe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"
Sep 16 17:48:35.603407 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.603235    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d23aec2-d092-4772-bd17-d7dff65c53fe-cabundle-cert\") pod \"auth-invalid-token-test-kserve-5fc974f796-zfp58\" (UID: \"3d23aec2-d092-4772-bd17-d7dff65c53fe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"
Sep 16 17:48:35.603407 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.603357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d23aec2-d092-4772-bd17-d7dff65c53fe-tls-certs\") pod \"auth-invalid-token-test-kserve-5fc974f796-zfp58\" (UID: \"3d23aec2-d092-4772-bd17-d7dff65c53fe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"
Sep 16 17:48:35.603475 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.603415    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5grhz\" (UniqueName: \"kubernetes.io/projected/3d23aec2-d092-4772-bd17-d7dff65c53fe-kube-api-access-5grhz\") pod \"auth-invalid-token-test-kserve-5fc974f796-zfp58\" (UID: \"3d23aec2-d092-4772-bd17-d7dff65c53fe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"
Sep 16 17:48:35.603475 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.603447    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d23aec2-d092-4772-bd17-d7dff65c53fe-tmp-dir\") pod \"auth-invalid-token-test-kserve-5fc974f796-zfp58\" (UID: \"3d23aec2-d092-4772-bd17-d7dff65c53fe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"
Sep 16 17:48:35.603674 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.603562    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d23aec2-d092-4772-bd17-d7dff65c53fe-home\") pod \"auth-invalid-token-test-kserve-5fc974f796-zfp58\" (UID: \"3d23aec2-d092-4772-bd17-d7dff65c53fe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"
Sep 16 17:48:35.603674 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.603610    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d23aec2-d092-4772-bd17-d7dff65c53fe-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5fc974f796-zfp58\" (UID: \"3d23aec2-d092-4772-bd17-d7dff65c53fe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"
Sep 16 17:48:35.603674 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.603669    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d23aec2-d092-4772-bd17-d7dff65c53fe-model-cache\") pod \"auth-invalid-token-test-kserve-5fc974f796-zfp58\" (UID: \"3d23aec2-d092-4772-bd17-d7dff65c53fe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"
Sep 16 17:48:35.603948 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.603880    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d23aec2-d092-4772-bd17-d7dff65c53fe-tmp-dir\") pod \"auth-invalid-token-test-kserve-5fc974f796-zfp58\" (UID: \"3d23aec2-d092-4772-bd17-d7dff65c53fe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"
Sep 16 17:48:35.603948 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.603908    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d23aec2-d092-4772-bd17-d7dff65c53fe-home\") pod \"auth-invalid-token-test-kserve-5fc974f796-zfp58\" (UID: \"3d23aec2-d092-4772-bd17-d7dff65c53fe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"
Sep 16 17:48:35.603948 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.603943    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d23aec2-d092-4772-bd17-d7dff65c53fe-cabundle-cert\") pod \"auth-invalid-token-test-kserve-5fc974f796-zfp58\" (UID: \"3d23aec2-d092-4772-bd17-d7dff65c53fe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"
Sep 16 17:48:35.604105 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.603970    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d23aec2-d092-4772-bd17-d7dff65c53fe-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-5fc974f796-zfp58\" (UID: \"3d23aec2-d092-4772-bd17-d7dff65c53fe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"
Sep 16 17:48:35.605727 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.605699    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d23aec2-d092-4772-bd17-d7dff65c53fe-dshm\") pod \"auth-invalid-token-test-kserve-5fc974f796-zfp58\" (UID: \"3d23aec2-d092-4772-bd17-d7dff65c53fe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"
Sep 16 17:48:35.605924 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.605908    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d23aec2-d092-4772-bd17-d7dff65c53fe-tls-certs\") pod \"auth-invalid-token-test-kserve-5fc974f796-zfp58\" (UID: \"3d23aec2-d092-4772-bd17-d7dff65c53fe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"
Sep 16 17:48:35.614359 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.614335    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5grhz\" (UniqueName: \"kubernetes.io/projected/3d23aec2-d092-4772-bd17-d7dff65c53fe-kube-api-access-5grhz\") pod \"auth-invalid-token-test-kserve-5fc974f796-zfp58\" (UID: \"3d23aec2-d092-4772-bd17-d7dff65c53fe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"
Sep 16 17:48:35.639561 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.639484    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:48:35.679023 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.678992    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"
Sep 16 17:48:35.918587 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.918495    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:48:35.967908 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:35.967883    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"]
Sep 16 17:48:35.969581 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:48:35.969553    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d23aec2_d092_4772_bd17_d7dff65c53fe.slice/crio-446769e1e6c38cc91776f6f65b0f29e1ba30a7c5b1a6e0c2f896478a53e9fabf WatchSource:0}: Error finding container 446769e1e6c38cc91776f6f65b0f29e1ba30a7c5b1a6e0c2f896478a53e9fabf: Status 404 returned error can't find the container with id 446769e1e6c38cc91776f6f65b0f29e1ba30a7c5b1a6e0c2f896478a53e9fabf
Sep 16 17:48:36.116224 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:36.116184    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58" event={"ID":"3d23aec2-d092-4772-bd17-d7dff65c53fe","Type":"ContainerStarted","Data":"cafe1cf10cba91f9eb81696ee49e947e8ca10a31ae6b482a900605c07aa1b817"}
Sep 16 17:48:36.116385 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:36.116248    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58" event={"ID":"3d23aec2-d092-4772-bd17-d7dff65c53fe","Type":"ContainerStarted","Data":"446769e1e6c38cc91776f6f65b0f29e1ba30a7c5b1a6e0c2f896478a53e9fabf"}
Sep 16 17:48:39.135601 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:39.135567    2577 generic.go:356] "Generic (PLEG): container finished" podID="3d23aec2-d092-4772-bd17-d7dff65c53fe" containerID="cafe1cf10cba91f9eb81696ee49e947e8ca10a31ae6b482a900605c07aa1b817" exitCode=0
Sep 16 17:48:39.135991 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:39.135642    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58" event={"ID":"3d23aec2-d092-4772-bd17-d7dff65c53fe","Type":"ContainerDied","Data":"cafe1cf10cba91f9eb81696ee49e947e8ca10a31ae6b482a900605c07aa1b817"}
Sep 16 17:48:40.143824 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:40.143789    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58" event={"ID":"3d23aec2-d092-4772-bd17-d7dff65c53fe","Type":"ContainerStarted","Data":"0f668cc241be27f329df77e026338d81b7d1a4a0489b6a6b6bdaa26c1d0c94db"}
Sep 16 17:48:40.186694 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:40.186632    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58" podStartSLOduration=2.949873857 podStartE2EDuration="5.186612989s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.236739132s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:48:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:48:40.18411467 +0000 UTC m=+1441.546793457" watchObservedRunningTime="2026-09-16 17:48:40.186612989 +0000 UTC m=+1441.549291775"
Sep 16 17:48:40.679593 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:40.679555    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"
Sep 16 17:48:42.827576 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:42.827509    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:48:42.946582 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:42.946543    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:48:43.146179 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:43.146097    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:48:45.491494 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:45.491453    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:48:45.599627 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:45.599593    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:48:45.679628 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:45.679571    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"
Sep 16 17:48:45.681175 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:45.681142    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58" podUID="3d23aec2-d092-4772-bd17-d7dff65c53fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Sep 16 17:48:52.685561 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:52.685525    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f"]
Sep 16 17:48:52.689384 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:52.689360    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f"
Sep 16 17:48:52.695068 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:52.695046    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 16 17:48:52.695375 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:52.695357    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 16 17:48:52.695479 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:52.695396    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-l5vjz\""
Sep 16 17:48:52.701944 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:52.701928    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 16 17:48:52.717797 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:52.717767    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f"]
Sep 16 17:48:52.763760 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:52.763728    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99feab73-35e7-48b2-84ef-06eff9b35a37-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f\" (UID: \"99feab73-35e7-48b2-84ef-06eff9b35a37\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f"
Sep 16 17:48:52.763948 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:52.763771    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp5fv\" (UniqueName: \"kubernetes.io/projected/99feab73-35e7-48b2-84ef-06eff9b35a37-kube-api-access-jp5fv\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f\" (UID: \"99feab73-35e7-48b2-84ef-06eff9b35a37\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f"
Sep 16 17:48:52.763948 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:52.763793    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99feab73-35e7-48b2-84ef-06eff9b35a37-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f\" (UID: \"99feab73-35e7-48b2-84ef-06eff9b35a37\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f"
Sep 16 17:48:52.763948 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:52.763854    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99feab73-35e7-48b2-84ef-06eff9b35a37-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f\" (UID: \"99feab73-35e7-48b2-84ef-06eff9b35a37\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f"
Sep 16 17:48:52.763948 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:52.763922    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99feab73-35e7-48b2-84ef-06eff9b35a37-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f\" (UID: \"99feab73-35e7-48b2-84ef-06eff9b35a37\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f"
Sep 16 17:48:52.764084 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:52.763979    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99feab73-35e7-48b2-84ef-06eff9b35a37-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f\" (UID: \"99feab73-35e7-48b2-84ef-06eff9b35a37\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f"
Sep 16 17:48:52.821340 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:52.821302    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:48:52.865001 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:52.864968    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99feab73-35e7-48b2-84ef-06eff9b35a37-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f\" (UID: \"99feab73-35e7-48b2-84ef-06eff9b35a37\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f"
Sep 16 17:48:52.865205 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:52.865029    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99feab73-35e7-48b2-84ef-06eff9b35a37-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f\" (UID: \"99feab73-35e7-48b2-84ef-06eff9b35a37\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f"
Sep 16 17:48:52.865205 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:52.865054    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp5fv\" (UniqueName: \"kubernetes.io/projected/99feab73-35e7-48b2-84ef-06eff9b35a37-kube-api-access-jp5fv\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f\" (UID: \"99feab73-35e7-48b2-84ef-06eff9b35a37\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f"
Sep 16 17:48:52.865205 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:52.865074    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99feab73-35e7-48b2-84ef-06eff9b35a37-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f\" (UID: \"99feab73-35e7-48b2-84ef-06eff9b35a37\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f"
Sep 16 17:48:52.865205 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:52.865101    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99feab73-35e7-48b2-84ef-06eff9b35a37-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f\" (UID: \"99feab73-35e7-48b2-84ef-06eff9b35a37\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f"
Sep 16 17:48:52.865427 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:52.865258    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99feab73-35e7-48b2-84ef-06eff9b35a37-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f\" (UID: \"99feab73-35e7-48b2-84ef-06eff9b35a37\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f"
Sep 16 17:48:52.865488 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:52.865453    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99feab73-35e7-48b2-84ef-06eff9b35a37-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f\" (UID: \"99feab73-35e7-48b2-84ef-06eff9b35a37\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f"
Sep 16 17:48:52.865547 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:52.865508    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99feab73-35e7-48b2-84ef-06eff9b35a37-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f\" (UID: \"99feab73-35e7-48b2-84ef-06eff9b35a37\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f"
Sep 16 17:48:52.865547 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:52.865529    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99feab73-35e7-48b2-84ef-06eff9b35a37-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f\" (UID: \"99feab73-35e7-48b2-84ef-06eff9b35a37\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f"
Sep 16 17:48:52.867446 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:52.867418    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99feab73-35e7-48b2-84ef-06eff9b35a37-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f\" (UID: \"99feab73-35e7-48b2-84ef-06eff9b35a37\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f"
Sep 16 17:48:52.867711 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:52.867693    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99feab73-35e7-48b2-84ef-06eff9b35a37-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f\" (UID: \"99feab73-35e7-48b2-84ef-06eff9b35a37\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f"
Sep 16 17:48:52.879965 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:52.879943    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp5fv\" (UniqueName: \"kubernetes.io/projected/99feab73-35e7-48b2-84ef-06eff9b35a37-kube-api-access-jp5fv\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f\" (UID: \"99feab73-35e7-48b2-84ef-06eff9b35a37\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f"
Sep 16 17:48:53.003622 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:53.003533    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f"
Sep 16 17:48:53.264339 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:53.264252    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:48:53.311579 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:53.311550    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f"]
Sep 16 17:48:53.315121 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:48:53.315090    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99feab73_35e7_48b2_84ef_06eff9b35a37.slice/crio-711a230c4e6cd68306c2e215a68904a62f2716b6ddccdf25076e4753ef1e1ede WatchSource:0}: Error finding container 711a230c4e6cd68306c2e215a68904a62f2716b6ddccdf25076e4753ef1e1ede: Status 404 returned error can't find the container with id 711a230c4e6cd68306c2e215a68904a62f2716b6ddccdf25076e4753ef1e1ede
Sep 16 17:48:54.234161 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:54.234126    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f" event={"ID":"99feab73-35e7-48b2-84ef-06eff9b35a37","Type":"ContainerStarted","Data":"e223e4aa9219161b0be89642e014ec33030b7902cedce2bcc174de29f7689aa3"}
Sep 16 17:48:54.234517 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:54.234184    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f" event={"ID":"99feab73-35e7-48b2-84ef-06eff9b35a37","Type":"ContainerStarted","Data":"711a230c4e6cd68306c2e215a68904a62f2716b6ddccdf25076e4753ef1e1ede"}
Sep 16 17:48:54.277061 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:54.277008    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f" podStartSLOduration=2.276992259 podStartE2EDuration="2.276992259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:48:54.27397654 +0000 UTC m=+1455.636660626" watchObservedRunningTime="2026-09-16 17:48:54.276992259 +0000 UTC m=+1455.639671045"
Sep 16 17:48:54.489914 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:54.489803    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:48:55.004237 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:55.004199    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f"
Sep 16 17:48:55.679980 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:48:55.679938    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58" podUID="3d23aec2-d092-4772-bd17-d7dff65c53fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Sep 16 17:49:00.759269 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:00.759228    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:49:00.842028 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:00.841985    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:49:03.003717 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:03.003670    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f"
Sep 16 17:49:03.016603 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:03.016572    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f"
Sep 16 17:49:03.296875 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:03.296783    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f"
Sep 16 17:49:05.679918 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:05.679876    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58" podUID="3d23aec2-d092-4772-bd17-d7dff65c53fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Sep 16 17:49:05.906620 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:05.906582    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:49:15.679488 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:15.679439    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58" podUID="3d23aec2-d092-4772-bd17-d7dff65c53fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Sep 16 17:49:25.680287 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:25.680244    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58" podUID="3d23aec2-d092-4772-bd17-d7dff65c53fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Sep 16 17:49:27.558203 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:27.558136    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:49:27.603079 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:27.603035    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f"]
Sep 16 17:49:27.612257 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:49:27.612206    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 17:49:27.612257 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:49:27.612257 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 17:49:27.612257 ip-10-0-136-122 kubenswrapper[2577]:  > containerID="e223e4aa9219161b0be89642e014ec33030b7902cedce2bcc174de29f7689aa3" cmd=["/bin/sleep","15"]
Sep 16 17:49:27.612562 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:27.612273    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f" message="" err=<
Sep 16 17:49:27.612562 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:49:27.612562 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 17:49:27.612562 ip-10-0-136-122 kubenswrapper[2577]:  >
Sep 16 17:49:27.612562 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:49:27.612291    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 17:49:27.612562 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:49:27.612562 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 17:49:27.612562 ip-10-0-136-122 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f" podUID="99feab73-35e7-48b2-84ef-06eff9b35a37" containerName="main" containerID="cri-o://e223e4aa9219161b0be89642e014ec33030b7902cedce2bcc174de29f7689aa3"
Sep 16 17:49:27.612562 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:27.612350    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f" podUID="99feab73-35e7-48b2-84ef-06eff9b35a37" containerName="main" containerID="cri-o://e223e4aa9219161b0be89642e014ec33030b7902cedce2bcc174de29f7689aa3" gracePeriod=60
Sep 16 17:49:27.703994 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:27.703954    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:49:28.358161 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:28.358135    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f"
Sep 16 17:49:28.430314 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:28.430276    2577 generic.go:356] "Generic (PLEG): container finished" podID="99feab73-35e7-48b2-84ef-06eff9b35a37" containerID="e223e4aa9219161b0be89642e014ec33030b7902cedce2bcc174de29f7689aa3" exitCode=0
Sep 16 17:49:28.430314 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:28.430316    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f"
Sep 16 17:49:28.430583 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:28.430346    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f" event={"ID":"99feab73-35e7-48b2-84ef-06eff9b35a37","Type":"ContainerDied","Data":"e223e4aa9219161b0be89642e014ec33030b7902cedce2bcc174de29f7689aa3"}
Sep 16 17:49:28.430583 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:28.430402    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f" event={"ID":"99feab73-35e7-48b2-84ef-06eff9b35a37","Type":"ContainerDied","Data":"711a230c4e6cd68306c2e215a68904a62f2716b6ddccdf25076e4753ef1e1ede"}
Sep 16 17:49:28.430583 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:28.430432    2577 scope.go:117] "RemoveContainer" containerID="e223e4aa9219161b0be89642e014ec33030b7902cedce2bcc174de29f7689aa3"
Sep 16 17:49:28.440457 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:28.440439    2577 scope.go:117] "RemoveContainer" containerID="e223e4aa9219161b0be89642e014ec33030b7902cedce2bcc174de29f7689aa3"
Sep 16 17:49:28.440785 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:49:28.440761    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e223e4aa9219161b0be89642e014ec33030b7902cedce2bcc174de29f7689aa3\": container with ID starting with e223e4aa9219161b0be89642e014ec33030b7902cedce2bcc174de29f7689aa3 not found: ID does not exist" containerID="e223e4aa9219161b0be89642e014ec33030b7902cedce2bcc174de29f7689aa3"
Sep 16 17:49:28.440855 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:28.440797    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e223e4aa9219161b0be89642e014ec33030b7902cedce2bcc174de29f7689aa3"} err="failed to get container status \"e223e4aa9219161b0be89642e014ec33030b7902cedce2bcc174de29f7689aa3\": rpc error: code = NotFound desc = could not find container \"e223e4aa9219161b0be89642e014ec33030b7902cedce2bcc174de29f7689aa3\": container with ID starting with e223e4aa9219161b0be89642e014ec33030b7902cedce2bcc174de29f7689aa3 not found: ID does not exist"
Sep 16 17:49:28.490911 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:28.490807    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jp5fv\" (UniqueName: \"kubernetes.io/projected/99feab73-35e7-48b2-84ef-06eff9b35a37-kube-api-access-jp5fv\") pod \"99feab73-35e7-48b2-84ef-06eff9b35a37\" (UID: \"99feab73-35e7-48b2-84ef-06eff9b35a37\") "
Sep 16 17:49:28.490911 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:28.490851    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99feab73-35e7-48b2-84ef-06eff9b35a37-dshm\") pod \"99feab73-35e7-48b2-84ef-06eff9b35a37\" (UID: \"99feab73-35e7-48b2-84ef-06eff9b35a37\") "
Sep 16 17:49:28.491117 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:28.490943    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99feab73-35e7-48b2-84ef-06eff9b35a37-model-cache\") pod \"99feab73-35e7-48b2-84ef-06eff9b35a37\" (UID: \"99feab73-35e7-48b2-84ef-06eff9b35a37\") "
Sep 16 17:49:28.491117 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:28.491022    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99feab73-35e7-48b2-84ef-06eff9b35a37-home\") pod \"99feab73-35e7-48b2-84ef-06eff9b35a37\" (UID: \"99feab73-35e7-48b2-84ef-06eff9b35a37\") "
Sep 16 17:49:28.491117 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:28.491049    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99feab73-35e7-48b2-84ef-06eff9b35a37-tls-certs\") pod \"99feab73-35e7-48b2-84ef-06eff9b35a37\" (UID: \"99feab73-35e7-48b2-84ef-06eff9b35a37\") "
Sep 16 17:49:28.491117 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:28.491102    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99feab73-35e7-48b2-84ef-06eff9b35a37-tmp-dir\") pod \"99feab73-35e7-48b2-84ef-06eff9b35a37\" (UID: \"99feab73-35e7-48b2-84ef-06eff9b35a37\") "
Sep 16 17:49:28.491347 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:28.491246    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99feab73-35e7-48b2-84ef-06eff9b35a37-model-cache" (OuterVolumeSpecName: "model-cache") pod "99feab73-35e7-48b2-84ef-06eff9b35a37" (UID: "99feab73-35e7-48b2-84ef-06eff9b35a37"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:49:28.491400 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:28.491362    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99feab73-35e7-48b2-84ef-06eff9b35a37-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:49:28.491507 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:28.491476    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99feab73-35e7-48b2-84ef-06eff9b35a37-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "99feab73-35e7-48b2-84ef-06eff9b35a37" (UID: "99feab73-35e7-48b2-84ef-06eff9b35a37"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:49:28.491507 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:28.491496    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99feab73-35e7-48b2-84ef-06eff9b35a37-home" (OuterVolumeSpecName: "home") pod "99feab73-35e7-48b2-84ef-06eff9b35a37" (UID: "99feab73-35e7-48b2-84ef-06eff9b35a37"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:49:28.493201 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:28.493178    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99feab73-35e7-48b2-84ef-06eff9b35a37-dshm" (OuterVolumeSpecName: "dshm") pod "99feab73-35e7-48b2-84ef-06eff9b35a37" (UID: "99feab73-35e7-48b2-84ef-06eff9b35a37"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:49:28.493291 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:28.493260    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99feab73-35e7-48b2-84ef-06eff9b35a37-kube-api-access-jp5fv" (OuterVolumeSpecName: "kube-api-access-jp5fv") pod "99feab73-35e7-48b2-84ef-06eff9b35a37" (UID: "99feab73-35e7-48b2-84ef-06eff9b35a37"). InnerVolumeSpecName "kube-api-access-jp5fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:49:28.493951 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:28.493927    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99feab73-35e7-48b2-84ef-06eff9b35a37-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "99feab73-35e7-48b2-84ef-06eff9b35a37" (UID: "99feab73-35e7-48b2-84ef-06eff9b35a37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:49:28.592042 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:28.592006    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp5fv\" (UniqueName: \"kubernetes.io/projected/99feab73-35e7-48b2-84ef-06eff9b35a37-kube-api-access-jp5fv\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:49:28.592042 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:28.592040    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99feab73-35e7-48b2-84ef-06eff9b35a37-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:49:28.592042 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:28.592052    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99feab73-35e7-48b2-84ef-06eff9b35a37-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:49:28.592481 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:28.592063    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99feab73-35e7-48b2-84ef-06eff9b35a37-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:49:28.592481 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:28.592075    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99feab73-35e7-48b2-84ef-06eff9b35a37-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:49:28.766570 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:28.766534    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f"]
Sep 16 17:49:28.770296 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:28.770273    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f"]
Sep 16 17:49:29.081642 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:29.081606    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99feab73-35e7-48b2-84ef-06eff9b35a37" path="/var/lib/kubelet/pods/99feab73-35e7-48b2-84ef-06eff9b35a37/volumes"
Sep 16 17:49:29.283715 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:29.283669    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqhj5f" podUID="99feab73-35e7-48b2-84ef-06eff9b35a37" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded"
Sep 16 17:49:35.680049 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:35.680008    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58" podUID="3d23aec2-d092-4772-bd17-d7dff65c53fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Sep 16 17:49:45.680270 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:45.680220    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58" podUID="3d23aec2-d092-4772-bd17-d7dff65c53fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Sep 16 17:49:55.680266 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:49:55.680222    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58" podUID="3d23aec2-d092-4772-bd17-d7dff65c53fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Sep 16 17:50:05.679587 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:05.679481    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58" podUID="3d23aec2-d092-4772-bd17-d7dff65c53fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Sep 16 17:50:15.680245 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:15.680198    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58" podUID="3d23aec2-d092-4772-bd17-d7dff65c53fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Sep 16 17:50:17.407511 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:17.407475    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:50:17.539678 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:17.539639    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:50:22.165419 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:22.165385    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg"]
Sep 16 17:50:22.165937 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:22.165827    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99feab73-35e7-48b2-84ef-06eff9b35a37" containerName="main"
Sep 16 17:50:22.165937 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:22.165841    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="99feab73-35e7-48b2-84ef-06eff9b35a37" containerName="main"
Sep 16 17:50:22.165937 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:22.165922    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="99feab73-35e7-48b2-84ef-06eff9b35a37" containerName="main"
Sep 16 17:50:22.172173 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:22.172152    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg"
Sep 16 17:50:22.174901 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:22.174845    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 16 17:50:22.176098 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:22.176072    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 16 17:50:22.176221 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:22.176115    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-bxcw4\""
Sep 16 17:50:22.176281 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:22.176097    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 16 17:50:22.185751 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:22.185727    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg"]
Sep 16 17:50:22.204301 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:22.204243    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:50:22.262700 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:22.262663    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e838854-64ce-4e48-a97d-8e4a68f193ad-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg\" (UID: \"6e838854-64ce-4e48-a97d-8e4a68f193ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg"
Sep 16 17:50:22.262923 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:22.262711    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e838854-64ce-4e48-a97d-8e4a68f193ad-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg\" (UID: \"6e838854-64ce-4e48-a97d-8e4a68f193ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg"
Sep 16 17:50:22.262923 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:22.262735    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e838854-64ce-4e48-a97d-8e4a68f193ad-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg\" (UID: \"6e838854-64ce-4e48-a97d-8e4a68f193ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg"
Sep 16 17:50:22.262923 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:22.262784    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcmlp\" (UniqueName: \"kubernetes.io/projected/6e838854-64ce-4e48-a97d-8e4a68f193ad-kube-api-access-zcmlp\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg\" (UID: \"6e838854-64ce-4e48-a97d-8e4a68f193ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg"
Sep 16 17:50:22.262923 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:22.262805    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e838854-64ce-4e48-a97d-8e4a68f193ad-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg\" (UID: \"6e838854-64ce-4e48-a97d-8e4a68f193ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg"
Sep 16 17:50:22.263119 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:22.262916    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e838854-64ce-4e48-a97d-8e4a68f193ad-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg\" (UID: \"6e838854-64ce-4e48-a97d-8e4a68f193ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg"
Sep 16 17:50:22.363847 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:22.363805    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e838854-64ce-4e48-a97d-8e4a68f193ad-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg\" (UID: \"6e838854-64ce-4e48-a97d-8e4a68f193ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg"
Sep 16 17:50:22.364071 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:22.363859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e838854-64ce-4e48-a97d-8e4a68f193ad-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg\" (UID: \"6e838854-64ce-4e48-a97d-8e4a68f193ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg"
Sep 16 17:50:22.364071 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:22.363904    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e838854-64ce-4e48-a97d-8e4a68f193ad-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg\" (UID: \"6e838854-64ce-4e48-a97d-8e4a68f193ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg"
Sep 16 17:50:22.364071 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:22.363925    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcmlp\" (UniqueName: \"kubernetes.io/projected/6e838854-64ce-4e48-a97d-8e4a68f193ad-kube-api-access-zcmlp\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg\" (UID: \"6e838854-64ce-4e48-a97d-8e4a68f193ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg"
Sep 16 17:50:22.364071 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:22.363947    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e838854-64ce-4e48-a97d-8e4a68f193ad-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg\" (UID: \"6e838854-64ce-4e48-a97d-8e4a68f193ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg"
Sep 16 17:50:22.364071 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:22.363994    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e838854-64ce-4e48-a97d-8e4a68f193ad-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg\" (UID: \"6e838854-64ce-4e48-a97d-8e4a68f193ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg"
Sep 16 17:50:22.364470 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:22.364437    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e838854-64ce-4e48-a97d-8e4a68f193ad-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg\" (UID: \"6e838854-64ce-4e48-a97d-8e4a68f193ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg"
Sep 16 17:50:22.364598 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:22.364474    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e838854-64ce-4e48-a97d-8e4a68f193ad-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg\" (UID: \"6e838854-64ce-4e48-a97d-8e4a68f193ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg"
Sep 16 17:50:22.364598 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:22.364499    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e838854-64ce-4e48-a97d-8e4a68f193ad-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg\" (UID: \"6e838854-64ce-4e48-a97d-8e4a68f193ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg"
Sep 16 17:50:22.366447 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:22.366421    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e838854-64ce-4e48-a97d-8e4a68f193ad-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg\" (UID: \"6e838854-64ce-4e48-a97d-8e4a68f193ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg"
Sep 16 17:50:22.366592 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:22.366577    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e838854-64ce-4e48-a97d-8e4a68f193ad-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg\" (UID: \"6e838854-64ce-4e48-a97d-8e4a68f193ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg"
Sep 16 17:50:22.382507 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:22.382474    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcmlp\" (UniqueName: \"kubernetes.io/projected/6e838854-64ce-4e48-a97d-8e4a68f193ad-kube-api-access-zcmlp\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg\" (UID: \"6e838854-64ce-4e48-a97d-8e4a68f193ad\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg"
Sep 16 17:50:22.447666 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:22.447563    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:50:22.483720 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:22.483691    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg"
Sep 16 17:50:22.770784 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:22.770757    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg"]
Sep 16 17:50:22.772308 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:50:22.772281    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e838854_64ce_4e48_a97d_8e4a68f193ad.slice/crio-9bccd9488d4408e3f4a5c946c17dee2082570ecadba9ae4db32920bd56d34c9e WatchSource:0}: Error finding container 9bccd9488d4408e3f4a5c946c17dee2082570ecadba9ae4db32920bd56d34c9e: Status 404 returned error can't find the container with id 9bccd9488d4408e3f4a5c946c17dee2082570ecadba9ae4db32920bd56d34c9e
Sep 16 17:50:23.744488 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:23.744450    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg" event={"ID":"6e838854-64ce-4e48-a97d-8e4a68f193ad","Type":"ContainerStarted","Data":"aa1463880aeebe1a21805d5f9415f6130d1f478e1fe97434d89fa3f3ce2d74ea"}
Sep 16 17:50:23.744996 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:23.744503    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg" event={"ID":"6e838854-64ce-4e48-a97d-8e4a68f193ad","Type":"ContainerStarted","Data":"9bccd9488d4408e3f4a5c946c17dee2082570ecadba9ae4db32920bd56d34c9e"}
Sep 16 17:50:23.770011 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:23.769951    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg" podStartSLOduration=1.769930437 podStartE2EDuration="1.769930437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:50:23.767503948 +0000 UTC m=+1545.130182735" watchObservedRunningTime="2026-09-16 17:50:23.769930437 +0000 UTC m=+1545.132609224"
Sep 16 17:50:24.484196 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:24.484163    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg"
Sep 16 17:50:24.770990 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:24.770902    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:50:25.679957 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:25.679919    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58" podUID="3d23aec2-d092-4772-bd17-d7dff65c53fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Sep 16 17:50:25.926619 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:25.926564    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:50:25.987638 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:25.987550    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:50:27.508031 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:27.507987    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:50:30.830482 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:30.830446    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:50:30.932472 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:30.932439    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:50:32.484907 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:32.484875    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg"
Sep 16 17:50:32.497184 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:32.497154    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg"
Sep 16 17:50:32.805911 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:32.805813    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg"
Sep 16 17:50:35.690145 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:35.690113    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"
Sep 16 17:50:35.697952 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:35.697927    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"
Sep 16 17:50:37.288139 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:37.288105    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"]
Sep 16 17:50:37.303668 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:37.303626    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:50:37.404443 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:37.404410    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:50:52.312141 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:52.312077    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58" podUID="3d23aec2-d092-4772-bd17-d7dff65c53fe" containerName="main" containerID="cri-o://0f668cc241be27f329df77e026338d81b7d1a4a0489b6a6b6bdaa26c1d0c94db" gracePeriod=45
Sep 16 17:50:52.690910 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:52.690800    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58" podUID="3d23aec2-d092-4772-bd17-d7dff65c53fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Sep 16 17:50:52.903106 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:52.903076    2577 generic.go:356] "Generic (PLEG): container finished" podID="3d23aec2-d092-4772-bd17-d7dff65c53fe" containerID="0f668cc241be27f329df77e026338d81b7d1a4a0489b6a6b6bdaa26c1d0c94db" exitCode=0
Sep 16 17:50:52.903253 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:52.903155    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58" event={"ID":"3d23aec2-d092-4772-bd17-d7dff65c53fe","Type":"ContainerDied","Data":"0f668cc241be27f329df77e026338d81b7d1a4a0489b6a6b6bdaa26c1d0c94db"}
Sep 16 17:50:52.983559 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:52.983532    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"
Sep 16 17:50:53.040168 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:53.040136    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d23aec2-d092-4772-bd17-d7dff65c53fe-model-cache\") pod \"3d23aec2-d092-4772-bd17-d7dff65c53fe\" (UID: \"3d23aec2-d092-4772-bd17-d7dff65c53fe\") "
Sep 16 17:50:53.040168 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:53.040172    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d23aec2-d092-4772-bd17-d7dff65c53fe-home\") pod \"3d23aec2-d092-4772-bd17-d7dff65c53fe\" (UID: \"3d23aec2-d092-4772-bd17-d7dff65c53fe\") "
Sep 16 17:50:53.040413 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:53.040191    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d23aec2-d092-4772-bd17-d7dff65c53fe-kserve-provision-location\") pod \"3d23aec2-d092-4772-bd17-d7dff65c53fe\" (UID: \"3d23aec2-d092-4772-bd17-d7dff65c53fe\") "
Sep 16 17:50:53.040413 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:53.040208    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5grhz\" (UniqueName: \"kubernetes.io/projected/3d23aec2-d092-4772-bd17-d7dff65c53fe-kube-api-access-5grhz\") pod \"3d23aec2-d092-4772-bd17-d7dff65c53fe\" (UID: \"3d23aec2-d092-4772-bd17-d7dff65c53fe\") "
Sep 16 17:50:53.040413 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:53.040227    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d23aec2-d092-4772-bd17-d7dff65c53fe-dshm\") pod \"3d23aec2-d092-4772-bd17-d7dff65c53fe\" (UID: \"3d23aec2-d092-4772-bd17-d7dff65c53fe\") "
Sep 16 17:50:53.040413 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:53.040284    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d23aec2-d092-4772-bd17-d7dff65c53fe-tmp-dir\") pod \"3d23aec2-d092-4772-bd17-d7dff65c53fe\" (UID: \"3d23aec2-d092-4772-bd17-d7dff65c53fe\") "
Sep 16 17:50:53.040413 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:53.040321    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d23aec2-d092-4772-bd17-d7dff65c53fe-cabundle-cert\") pod \"3d23aec2-d092-4772-bd17-d7dff65c53fe\" (UID: \"3d23aec2-d092-4772-bd17-d7dff65c53fe\") "
Sep 16 17:50:53.040664 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:53.040419    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d23aec2-d092-4772-bd17-d7dff65c53fe-tls-certs\") pod \"3d23aec2-d092-4772-bd17-d7dff65c53fe\" (UID: \"3d23aec2-d092-4772-bd17-d7dff65c53fe\") "
Sep 16 17:50:53.040664 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:53.040522    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d23aec2-d092-4772-bd17-d7dff65c53fe-model-cache" (OuterVolumeSpecName: "model-cache") pod "3d23aec2-d092-4772-bd17-d7dff65c53fe" (UID: "3d23aec2-d092-4772-bd17-d7dff65c53fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:50:53.040774 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:53.040687    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d23aec2-d092-4772-bd17-d7dff65c53fe-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:50:53.041030 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:53.041009    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d23aec2-d092-4772-bd17-d7dff65c53fe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3d23aec2-d092-4772-bd17-d7dff65c53fe" (UID: "3d23aec2-d092-4772-bd17-d7dff65c53fe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:50:53.041117 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:53.041042    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d23aec2-d092-4772-bd17-d7dff65c53fe-home" (OuterVolumeSpecName: "home") pod "3d23aec2-d092-4772-bd17-d7dff65c53fe" (UID: "3d23aec2-d092-4772-bd17-d7dff65c53fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:50:53.042606 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:53.042582    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d23aec2-d092-4772-bd17-d7dff65c53fe-dshm" (OuterVolumeSpecName: "dshm") pod "3d23aec2-d092-4772-bd17-d7dff65c53fe" (UID: "3d23aec2-d092-4772-bd17-d7dff65c53fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:50:53.042886 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:53.042841    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d23aec2-d092-4772-bd17-d7dff65c53fe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d23aec2-d092-4772-bd17-d7dff65c53fe" (UID: "3d23aec2-d092-4772-bd17-d7dff65c53fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:50:53.042991 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:53.042844    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d23aec2-d092-4772-bd17-d7dff65c53fe-kube-api-access-5grhz" (OuterVolumeSpecName: "kube-api-access-5grhz") pod "3d23aec2-d092-4772-bd17-d7dff65c53fe" (UID: "3d23aec2-d092-4772-bd17-d7dff65c53fe"). InnerVolumeSpecName "kube-api-access-5grhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:50:53.059047 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:53.059016    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d23aec2-d092-4772-bd17-d7dff65c53fe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3d23aec2-d092-4772-bd17-d7dff65c53fe" (UID: "3d23aec2-d092-4772-bd17-d7dff65c53fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:50:53.069237 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:53.069211    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d23aec2-d092-4772-bd17-d7dff65c53fe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3d23aec2-d092-4772-bd17-d7dff65c53fe" (UID: "3d23aec2-d092-4772-bd17-d7dff65c53fe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:50:53.141210 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:53.141182    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d23aec2-d092-4772-bd17-d7dff65c53fe-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:50:53.141210 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:53.141208    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d23aec2-d092-4772-bd17-d7dff65c53fe-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:50:53.141365 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:53.141218    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d23aec2-d092-4772-bd17-d7dff65c53fe-kserve-provision-location\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:50:53.141365 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:53.141227    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5grhz\" (UniqueName: \"kubernetes.io/projected/3d23aec2-d092-4772-bd17-d7dff65c53fe-kube-api-access-5grhz\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:50:53.141365 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:53.141238    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d23aec2-d092-4772-bd17-d7dff65c53fe-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:50:53.141365 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:53.141250    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d23aec2-d092-4772-bd17-d7dff65c53fe-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:50:53.141365 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:53.141259    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d23aec2-d092-4772-bd17-d7dff65c53fe-cabundle-cert\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:50:53.909916 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:53.909883    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58" event={"ID":"3d23aec2-d092-4772-bd17-d7dff65c53fe","Type":"ContainerDied","Data":"446769e1e6c38cc91776f6f65b0f29e1ba30a7c5b1a6e0c2f896478a53e9fabf"}
Sep 16 17:50:53.910363 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:53.909946    2577 scope.go:117] "RemoveContainer" containerID="0f668cc241be27f329df77e026338d81b7d1a4a0489b6a6b6bdaa26c1d0c94db"
Sep 16 17:50:53.910363 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:53.909890    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"
Sep 16 17:50:53.919008 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:53.918986    2577 scope.go:117] "RemoveContainer" containerID="cafe1cf10cba91f9eb81696ee49e947e8ca10a31ae6b482a900605c07aa1b817"
Sep 16 17:50:53.932802 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:53.932779    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"]
Sep 16 17:50:53.936922 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:53.936900    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-5fc974f796-zfp58"]
Sep 16 17:50:55.081401 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:55.081366    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d23aec2-d092-4772-bd17-d7dff65c53fe" path="/var/lib/kubelet/pods/3d23aec2-d092-4772-bd17-d7dff65c53fe/volumes"
Sep 16 17:50:56.039638 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:56.039604    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg"]
Sep 16 17:50:56.048691 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:50:56.048655    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 17:50:56.048691 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:50:56.048691 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 17:50:56.048691 ip-10-0-136-122 kubenswrapper[2577]:  > containerID="aa1463880aeebe1a21805d5f9415f6130d1f478e1fe97434d89fa3f3ce2d74ea" cmd=["/bin/sleep","15"]
Sep 16 17:50:56.049025 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:56.048716    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg" message="" err=<
Sep 16 17:50:56.049025 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:50:56.049025 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 17:50:56.049025 ip-10-0-136-122 kubenswrapper[2577]:  >
Sep 16 17:50:56.049025 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:50:56.048735    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 17:50:56.049025 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:50:56.049025 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 17:50:56.049025 ip-10-0-136-122 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg" podUID="6e838854-64ce-4e48-a97d-8e4a68f193ad" containerName="main" containerID="cri-o://aa1463880aeebe1a21805d5f9415f6130d1f478e1fe97434d89fa3f3ce2d74ea"
Sep 16 17:50:56.049025 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:56.048777    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg" podUID="6e838854-64ce-4e48-a97d-8e4a68f193ad" containerName="main" containerID="cri-o://aa1463880aeebe1a21805d5f9415f6130d1f478e1fe97434d89fa3f3ce2d74ea" gracePeriod=60
Sep 16 17:50:56.052442 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:56.052397    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:50:56.134446 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:56.134407    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:50:56.406832 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:56.406808    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg"
Sep 16 17:50:56.470697 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:56.470661    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e838854-64ce-4e48-a97d-8e4a68f193ad-model-cache\") pod \"6e838854-64ce-4e48-a97d-8e4a68f193ad\" (UID: \"6e838854-64ce-4e48-a97d-8e4a68f193ad\") "
Sep 16 17:50:56.470890 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:56.470703    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e838854-64ce-4e48-a97d-8e4a68f193ad-dshm\") pod \"6e838854-64ce-4e48-a97d-8e4a68f193ad\" (UID: \"6e838854-64ce-4e48-a97d-8e4a68f193ad\") "
Sep 16 17:50:56.470890 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:56.470749    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e838854-64ce-4e48-a97d-8e4a68f193ad-tls-certs\") pod \"6e838854-64ce-4e48-a97d-8e4a68f193ad\" (UID: \"6e838854-64ce-4e48-a97d-8e4a68f193ad\") "
Sep 16 17:50:56.470890 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:56.470788    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcmlp\" (UniqueName: \"kubernetes.io/projected/6e838854-64ce-4e48-a97d-8e4a68f193ad-kube-api-access-zcmlp\") pod \"6e838854-64ce-4e48-a97d-8e4a68f193ad\" (UID: \"6e838854-64ce-4e48-a97d-8e4a68f193ad\") "
Sep 16 17:50:56.470890 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:56.470804    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e838854-64ce-4e48-a97d-8e4a68f193ad-home\") pod \"6e838854-64ce-4e48-a97d-8e4a68f193ad\" (UID: \"6e838854-64ce-4e48-a97d-8e4a68f193ad\") "
Sep 16 17:50:56.470890 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:56.470821    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e838854-64ce-4e48-a97d-8e4a68f193ad-tmp-dir\") pod \"6e838854-64ce-4e48-a97d-8e4a68f193ad\" (UID: \"6e838854-64ce-4e48-a97d-8e4a68f193ad\") "
Sep 16 17:50:56.471181 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:56.470993    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e838854-64ce-4e48-a97d-8e4a68f193ad-model-cache" (OuterVolumeSpecName: "model-cache") pod "6e838854-64ce-4e48-a97d-8e4a68f193ad" (UID: "6e838854-64ce-4e48-a97d-8e4a68f193ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:50:56.471181 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:56.471109    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e838854-64ce-4e48-a97d-8e4a68f193ad-home" (OuterVolumeSpecName: "home") pod "6e838854-64ce-4e48-a97d-8e4a68f193ad" (UID: "6e838854-64ce-4e48-a97d-8e4a68f193ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:50:56.471181 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:56.471116    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e838854-64ce-4e48-a97d-8e4a68f193ad-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:50:56.471181 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:56.471156    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e838854-64ce-4e48-a97d-8e4a68f193ad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6e838854-64ce-4e48-a97d-8e4a68f193ad" (UID: "6e838854-64ce-4e48-a97d-8e4a68f193ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:50:56.472976 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:56.472953    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e838854-64ce-4e48-a97d-8e4a68f193ad-dshm" (OuterVolumeSpecName: "dshm") pod "6e838854-64ce-4e48-a97d-8e4a68f193ad" (UID: "6e838854-64ce-4e48-a97d-8e4a68f193ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:50:56.473069 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:56.472996    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e838854-64ce-4e48-a97d-8e4a68f193ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6e838854-64ce-4e48-a97d-8e4a68f193ad" (UID: "6e838854-64ce-4e48-a97d-8e4a68f193ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:50:56.473069 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:56.473024    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e838854-64ce-4e48-a97d-8e4a68f193ad-kube-api-access-zcmlp" (OuterVolumeSpecName: "kube-api-access-zcmlp") pod "6e838854-64ce-4e48-a97d-8e4a68f193ad" (UID: "6e838854-64ce-4e48-a97d-8e4a68f193ad"). InnerVolumeSpecName "kube-api-access-zcmlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:50:56.571956 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:56.571833    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e838854-64ce-4e48-a97d-8e4a68f193ad-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:50:56.571956 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:56.571877    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e838854-64ce-4e48-a97d-8e4a68f193ad-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:50:56.571956 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:56.571891    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e838854-64ce-4e48-a97d-8e4a68f193ad-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:50:56.571956 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:56.571905    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcmlp\" (UniqueName: \"kubernetes.io/projected/6e838854-64ce-4e48-a97d-8e4a68f193ad-kube-api-access-zcmlp\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:50:56.571956 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:56.571915    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e838854-64ce-4e48-a97d-8e4a68f193ad-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:50:56.929616 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:56.929520    2577 generic.go:356] "Generic (PLEG): container finished" podID="6e838854-64ce-4e48-a97d-8e4a68f193ad" containerID="aa1463880aeebe1a21805d5f9415f6130d1f478e1fe97434d89fa3f3ce2d74ea" exitCode=0
Sep 16 17:50:56.929616 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:56.929570    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg"
Sep 16 17:50:56.929616 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:56.929603    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg" event={"ID":"6e838854-64ce-4e48-a97d-8e4a68f193ad","Type":"ContainerDied","Data":"aa1463880aeebe1a21805d5f9415f6130d1f478e1fe97434d89fa3f3ce2d74ea"}
Sep 16 17:50:56.929951 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:56.929661    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg" event={"ID":"6e838854-64ce-4e48-a97d-8e4a68f193ad","Type":"ContainerDied","Data":"9bccd9488d4408e3f4a5c946c17dee2082570ecadba9ae4db32920bd56d34c9e"}
Sep 16 17:50:56.929951 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:56.929685    2577 scope.go:117] "RemoveContainer" containerID="aa1463880aeebe1a21805d5f9415f6130d1f478e1fe97434d89fa3f3ce2d74ea"
Sep 16 17:50:56.938858 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:56.938839    2577 scope.go:117] "RemoveContainer" containerID="aa1463880aeebe1a21805d5f9415f6130d1f478e1fe97434d89fa3f3ce2d74ea"
Sep 16 17:50:56.939154 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:50:56.939135    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa1463880aeebe1a21805d5f9415f6130d1f478e1fe97434d89fa3f3ce2d74ea\": container with ID starting with aa1463880aeebe1a21805d5f9415f6130d1f478e1fe97434d89fa3f3ce2d74ea not found: ID does not exist" containerID="aa1463880aeebe1a21805d5f9415f6130d1f478e1fe97434d89fa3f3ce2d74ea"
Sep 16 17:50:56.939220 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:56.939165    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa1463880aeebe1a21805d5f9415f6130d1f478e1fe97434d89fa3f3ce2d74ea"} err="failed to get container status \"aa1463880aeebe1a21805d5f9415f6130d1f478e1fe97434d89fa3f3ce2d74ea\": rpc error: code = NotFound desc = could not find container \"aa1463880aeebe1a21805d5f9415f6130d1f478e1fe97434d89fa3f3ce2d74ea\": container with ID starting with aa1463880aeebe1a21805d5f9415f6130d1f478e1fe97434d89fa3f3ce2d74ea not found: ID does not exist"
Sep 16 17:50:56.954038 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:56.954006    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg"]
Sep 16 17:50:56.957819 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:56.957797    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jb5lg"]
Sep 16 17:50:57.086676 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:50:57.083334    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e838854-64ce-4e48-a97d-8e4a68f193ad" path="/var/lib/kubelet/pods/6e838854-64ce-4e48-a97d-8e4a68f193ad/volumes"
Sep 16 17:51:28.244816 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:28.244784    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:51:28.359606 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:28.359569    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:51:33.655672 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:33.655588    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:51:39.671220 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:39.671181    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:51:39.771151 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:39.771105    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:51:45.337758 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:45.337718    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:51:45.521703 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:45.521664    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:51:52.180024 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:52.179987    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:51:53.800937 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.800903    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"]
Sep 16 17:51:53.801341 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.801326    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d23aec2-d092-4772-bd17-d7dff65c53fe" containerName="main"
Sep 16 17:51:53.801387 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.801345    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d23aec2-d092-4772-bd17-d7dff65c53fe" containerName="main"
Sep 16 17:51:53.801387 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.801360    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d23aec2-d092-4772-bd17-d7dff65c53fe" containerName="storage-initializer"
Sep 16 17:51:53.801387 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.801367    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d23aec2-d092-4772-bd17-d7dff65c53fe" containerName="storage-initializer"
Sep 16 17:51:53.801387 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.801380    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e838854-64ce-4e48-a97d-8e4a68f193ad" containerName="main"
Sep 16 17:51:53.801387 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.801385    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e838854-64ce-4e48-a97d-8e4a68f193ad" containerName="main"
Sep 16 17:51:53.801555 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.801450    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e838854-64ce-4e48-a97d-8e4a68f193ad" containerName="main"
Sep 16 17:51:53.801555 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.801459    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d23aec2-d092-4772-bd17-d7dff65c53fe" containerName="main"
Sep 16 17:51:53.804691 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.804670    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"
Sep 16 17:51:53.808838 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.808810    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 16 17:51:53.809216 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.809186    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 17:51:53.809330 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.809222    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Sep 16 17:51:53.809473 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.809452    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 16 17:51:53.809883 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.809846    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-sqbws\""
Sep 16 17:51:53.809986 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.809914    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 16 17:51:53.835916 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.835884    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"]
Sep 16 17:51:53.837181 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.837155    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/525daaaf-c2af-4bf8-bf27-d777c52a5147-dshm\") pod \"auth-disabled-test-kserve-7555c9bf99-njnhl\" (UID: \"525daaaf-c2af-4bf8-bf27-d777c52a5147\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"
Sep 16 17:51:53.837262 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.837196    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/525daaaf-c2af-4bf8-bf27-d777c52a5147-tmp-dir\") pod \"auth-disabled-test-kserve-7555c9bf99-njnhl\" (UID: \"525daaaf-c2af-4bf8-bf27-d777c52a5147\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"
Sep 16 17:51:53.837317 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.837269    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/525daaaf-c2af-4bf8-bf27-d777c52a5147-model-cache\") pod \"auth-disabled-test-kserve-7555c9bf99-njnhl\" (UID: \"525daaaf-c2af-4bf8-bf27-d777c52a5147\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"
Sep 16 17:51:53.837317 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.837294    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/525daaaf-c2af-4bf8-bf27-d777c52a5147-kserve-provision-location\") pod \"auth-disabled-test-kserve-7555c9bf99-njnhl\" (UID: \"525daaaf-c2af-4bf8-bf27-d777c52a5147\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"
Sep 16 17:51:53.837412 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.837357    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dk8f\" (UniqueName: \"kubernetes.io/projected/525daaaf-c2af-4bf8-bf27-d777c52a5147-kube-api-access-9dk8f\") pod \"auth-disabled-test-kserve-7555c9bf99-njnhl\" (UID: \"525daaaf-c2af-4bf8-bf27-d777c52a5147\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"
Sep 16 17:51:53.837482 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.837455    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/525daaaf-c2af-4bf8-bf27-d777c52a5147-home\") pod \"auth-disabled-test-kserve-7555c9bf99-njnhl\" (UID: \"525daaaf-c2af-4bf8-bf27-d777c52a5147\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"
Sep 16 17:51:53.837529 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.837499    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/525daaaf-c2af-4bf8-bf27-d777c52a5147-tls-certs\") pod \"auth-disabled-test-kserve-7555c9bf99-njnhl\" (UID: \"525daaaf-c2af-4bf8-bf27-d777c52a5147\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"
Sep 16 17:51:53.837585 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.837570    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/525daaaf-c2af-4bf8-bf27-d777c52a5147-cabundle-cert\") pod \"auth-disabled-test-kserve-7555c9bf99-njnhl\" (UID: \"525daaaf-c2af-4bf8-bf27-d777c52a5147\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"
Sep 16 17:51:53.897585 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.897547    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:51:53.938090 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.938055    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/525daaaf-c2af-4bf8-bf27-d777c52a5147-cabundle-cert\") pod \"auth-disabled-test-kserve-7555c9bf99-njnhl\" (UID: \"525daaaf-c2af-4bf8-bf27-d777c52a5147\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"
Sep 16 17:51:53.938254 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.938107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/525daaaf-c2af-4bf8-bf27-d777c52a5147-dshm\") pod \"auth-disabled-test-kserve-7555c9bf99-njnhl\" (UID: \"525daaaf-c2af-4bf8-bf27-d777c52a5147\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"
Sep 16 17:51:53.938254 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.938131    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/525daaaf-c2af-4bf8-bf27-d777c52a5147-tmp-dir\") pod \"auth-disabled-test-kserve-7555c9bf99-njnhl\" (UID: \"525daaaf-c2af-4bf8-bf27-d777c52a5147\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"
Sep 16 17:51:53.938254 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.938168    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/525daaaf-c2af-4bf8-bf27-d777c52a5147-model-cache\") pod \"auth-disabled-test-kserve-7555c9bf99-njnhl\" (UID: \"525daaaf-c2af-4bf8-bf27-d777c52a5147\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"
Sep 16 17:51:53.938254 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.938189    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/525daaaf-c2af-4bf8-bf27-d777c52a5147-kserve-provision-location\") pod \"auth-disabled-test-kserve-7555c9bf99-njnhl\" (UID: \"525daaaf-c2af-4bf8-bf27-d777c52a5147\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"
Sep 16 17:51:53.938254 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.938244    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dk8f\" (UniqueName: \"kubernetes.io/projected/525daaaf-c2af-4bf8-bf27-d777c52a5147-kube-api-access-9dk8f\") pod \"auth-disabled-test-kserve-7555c9bf99-njnhl\" (UID: \"525daaaf-c2af-4bf8-bf27-d777c52a5147\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"
Sep 16 17:51:53.938509 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.938283    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/525daaaf-c2af-4bf8-bf27-d777c52a5147-home\") pod \"auth-disabled-test-kserve-7555c9bf99-njnhl\" (UID: \"525daaaf-c2af-4bf8-bf27-d777c52a5147\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"
Sep 16 17:51:53.938509 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.938307    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/525daaaf-c2af-4bf8-bf27-d777c52a5147-tls-certs\") pod \"auth-disabled-test-kserve-7555c9bf99-njnhl\" (UID: \"525daaaf-c2af-4bf8-bf27-d777c52a5147\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"
Sep 16 17:51:53.938674 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.938645    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/525daaaf-c2af-4bf8-bf27-d777c52a5147-tmp-dir\") pod \"auth-disabled-test-kserve-7555c9bf99-njnhl\" (UID: \"525daaaf-c2af-4bf8-bf27-d777c52a5147\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"
Sep 16 17:51:53.938734 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.938699    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/525daaaf-c2af-4bf8-bf27-d777c52a5147-kserve-provision-location\") pod \"auth-disabled-test-kserve-7555c9bf99-njnhl\" (UID: \"525daaaf-c2af-4bf8-bf27-d777c52a5147\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"
Sep 16 17:51:53.938778 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.938761    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/525daaaf-c2af-4bf8-bf27-d777c52a5147-model-cache\") pod \"auth-disabled-test-kserve-7555c9bf99-njnhl\" (UID: \"525daaaf-c2af-4bf8-bf27-d777c52a5147\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"
Sep 16 17:51:53.938934 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.938910    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/525daaaf-c2af-4bf8-bf27-d777c52a5147-home\") pod \"auth-disabled-test-kserve-7555c9bf99-njnhl\" (UID: \"525daaaf-c2af-4bf8-bf27-d777c52a5147\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"
Sep 16 17:51:53.939050 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.938989    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/525daaaf-c2af-4bf8-bf27-d777c52a5147-cabundle-cert\") pod \"auth-disabled-test-kserve-7555c9bf99-njnhl\" (UID: \"525daaaf-c2af-4bf8-bf27-d777c52a5147\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"
Sep 16 17:51:53.940447 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.940428    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/525daaaf-c2af-4bf8-bf27-d777c52a5147-dshm\") pod \"auth-disabled-test-kserve-7555c9bf99-njnhl\" (UID: \"525daaaf-c2af-4bf8-bf27-d777c52a5147\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"
Sep 16 17:51:53.940694 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.940678    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/525daaaf-c2af-4bf8-bf27-d777c52a5147-tls-certs\") pod \"auth-disabled-test-kserve-7555c9bf99-njnhl\" (UID: \"525daaaf-c2af-4bf8-bf27-d777c52a5147\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"
Sep 16 17:51:53.948111 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:53.948088    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dk8f\" (UniqueName: \"kubernetes.io/projected/525daaaf-c2af-4bf8-bf27-d777c52a5147-kube-api-access-9dk8f\") pod \"auth-disabled-test-kserve-7555c9bf99-njnhl\" (UID: \"525daaaf-c2af-4bf8-bf27-d777c52a5147\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"
Sep 16 17:51:54.117074 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:54.117038    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"
Sep 16 17:51:54.174079 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:54.174044    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:51:54.402061 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:54.402031    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"]
Sep 16 17:51:54.402818 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:51:54.402793    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod525daaaf_c2af_4bf8_bf27_d777c52a5147.slice/crio-4490c65b6e81512a3ad995c325dec9d81988ab8a7c387df36ae87faaceb4cd51 WatchSource:0}: Error finding container 4490c65b6e81512a3ad995c325dec9d81988ab8a7c387df36ae87faaceb4cd51: Status 404 returned error can't find the container with id 4490c65b6e81512a3ad995c325dec9d81988ab8a7c387df36ae87faaceb4cd51
Sep 16 17:51:55.237143 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:55.237103    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl" event={"ID":"525daaaf-c2af-4bf8-bf27-d777c52a5147","Type":"ContainerStarted","Data":"291eaad44d9b316ab0780c3a4a51f856d4a6a17ef5d64bef7fe33dc7319ff37b"}
Sep 16 17:51:55.237620 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:55.237158    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl" event={"ID":"525daaaf-c2af-4bf8-bf27-d777c52a5147","Type":"ContainerStarted","Data":"4490c65b6e81512a3ad995c325dec9d81988ab8a7c387df36ae87faaceb4cd51"}
Sep 16 17:51:57.252621 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:57.252584    2577 generic.go:356] "Generic (PLEG): container finished" podID="525daaaf-c2af-4bf8-bf27-d777c52a5147" containerID="291eaad44d9b316ab0780c3a4a51f856d4a6a17ef5d64bef7fe33dc7319ff37b" exitCode=0
Sep 16 17:51:57.253077 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:57.252659    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl" event={"ID":"525daaaf-c2af-4bf8-bf27-d777c52a5147","Type":"ContainerDied","Data":"291eaad44d9b316ab0780c3a4a51f856d4a6a17ef5d64bef7fe33dc7319ff37b"}
Sep 16 17:51:58.260922 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:58.260886    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl" event={"ID":"525daaaf-c2af-4bf8-bf27-d777c52a5147","Type":"ContainerStarted","Data":"3b622bb0ffb1b8265e6e8afec5c1be40c8958e8ff07b12d4adebb6a440a42d89"}
Sep 16 17:51:58.295214 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:58.295150    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl" podStartSLOduration=3.064312611 podStartE2EDuration="5.295130284s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.230817673s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:51:58.293246771 +0000 UTC m=+1639.655925557" watchObservedRunningTime="2026-09-16 17:51:58.295130284 +0000 UTC m=+1639.657809074"
Sep 16 17:51:58.761501 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:58.761464    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:51:58.851971 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:58.851934    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:51:59.117636 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:51:59.117596    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"
Sep 16 17:52:04.117683 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:04.117641    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"
Sep 16 17:52:04.119442 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:04.119400    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl" podUID="525daaaf-c2af-4bf8-bf27-d777c52a5147" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 16 17:52:08.553562 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.553526    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6"]
Sep 16 17:52:08.556718 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.556696    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6"
Sep 16 17:52:08.561067 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.561045    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 16 17:52:08.561936 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.561892    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-2ncbp\""
Sep 16 17:52:08.561936 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.561906    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 16 17:52:08.561936 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.561922    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 16 17:52:08.567027 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.567001    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:52:08.576177 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.576154    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6"]
Sep 16 17:52:08.671844 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.671807    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9032ed-5be7-4881-a09f-309c9dc48f3d-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mlst6\" (UID: \"7d9032ed-5be7-4881-a09f-309c9dc48f3d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6"
Sep 16 17:52:08.671844 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.671847    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d9032ed-5be7-4881-a09f-309c9dc48f3d-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mlst6\" (UID: \"7d9032ed-5be7-4881-a09f-309c9dc48f3d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6"
Sep 16 17:52:08.672069 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.671948    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9032ed-5be7-4881-a09f-309c9dc48f3d-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mlst6\" (UID: \"7d9032ed-5be7-4881-a09f-309c9dc48f3d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6"
Sep 16 17:52:08.672069 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.671984    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss2jj\" (UniqueName: \"kubernetes.io/projected/7d9032ed-5be7-4881-a09f-309c9dc48f3d-kube-api-access-ss2jj\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mlst6\" (UID: \"7d9032ed-5be7-4881-a09f-309c9dc48f3d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6"
Sep 16 17:52:08.672069 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.672014    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9032ed-5be7-4881-a09f-309c9dc48f3d-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mlst6\" (UID: \"7d9032ed-5be7-4881-a09f-309c9dc48f3d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6"
Sep 16 17:52:08.672172 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.672119    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9032ed-5be7-4881-a09f-309c9dc48f3d-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mlst6\" (UID: \"7d9032ed-5be7-4881-a09f-309c9dc48f3d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6"
Sep 16 17:52:08.773136 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.773101    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9032ed-5be7-4881-a09f-309c9dc48f3d-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mlst6\" (UID: \"7d9032ed-5be7-4881-a09f-309c9dc48f3d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6"
Sep 16 17:52:08.773136 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.773140    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d9032ed-5be7-4881-a09f-309c9dc48f3d-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mlst6\" (UID: \"7d9032ed-5be7-4881-a09f-309c9dc48f3d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6"
Sep 16 17:52:08.773370 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.773164    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9032ed-5be7-4881-a09f-309c9dc48f3d-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mlst6\" (UID: \"7d9032ed-5be7-4881-a09f-309c9dc48f3d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6"
Sep 16 17:52:08.773370 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.773193    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss2jj\" (UniqueName: \"kubernetes.io/projected/7d9032ed-5be7-4881-a09f-309c9dc48f3d-kube-api-access-ss2jj\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mlst6\" (UID: \"7d9032ed-5be7-4881-a09f-309c9dc48f3d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6"
Sep 16 17:52:08.773370 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.773230    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9032ed-5be7-4881-a09f-309c9dc48f3d-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mlst6\" (UID: \"7d9032ed-5be7-4881-a09f-309c9dc48f3d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6"
Sep 16 17:52:08.773370 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.773257    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9032ed-5be7-4881-a09f-309c9dc48f3d-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mlst6\" (UID: \"7d9032ed-5be7-4881-a09f-309c9dc48f3d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6"
Sep 16 17:52:08.773614 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.773584    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9032ed-5be7-4881-a09f-309c9dc48f3d-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mlst6\" (UID: \"7d9032ed-5be7-4881-a09f-309c9dc48f3d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6"
Sep 16 17:52:08.773799 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.773637    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d9032ed-5be7-4881-a09f-309c9dc48f3d-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mlst6\" (UID: \"7d9032ed-5be7-4881-a09f-309c9dc48f3d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6"
Sep 16 17:52:08.773922 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.773662    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9032ed-5be7-4881-a09f-309c9dc48f3d-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mlst6\" (UID: \"7d9032ed-5be7-4881-a09f-309c9dc48f3d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6"
Sep 16 17:52:08.775946 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.775921    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9032ed-5be7-4881-a09f-309c9dc48f3d-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mlst6\" (UID: \"7d9032ed-5be7-4881-a09f-309c9dc48f3d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6"
Sep 16 17:52:08.776321 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.776304    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9032ed-5be7-4881-a09f-309c9dc48f3d-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mlst6\" (UID: \"7d9032ed-5be7-4881-a09f-309c9dc48f3d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6"
Sep 16 17:52:08.783417 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.783382    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss2jj\" (UniqueName: \"kubernetes.io/projected/7d9032ed-5be7-4881-a09f-309c9dc48f3d-kube-api-access-ss2jj\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-mlst6\" (UID: \"7d9032ed-5be7-4881-a09f-309c9dc48f3d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6"
Sep 16 17:52:08.850275 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.850239    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h"]
Sep 16 17:52:08.853327 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.853309    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h"
Sep 16 17:52:08.856051 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.856027    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-vm4mf\""
Sep 16 17:52:08.866569 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.866541    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h"]
Sep 16 17:52:08.868826 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.868802    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6"
Sep 16 17:52:08.878084 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.877939    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:52:08.976403 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.976370    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rkmj\" (UniqueName: \"kubernetes.io/projected/3ad9133d-d3a8-420c-8ee2-ab7aff0c675d-kube-api-access-4rkmj\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h\" (UID: \"3ad9133d-d3a8-420c-8ee2-ab7aff0c675d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h"
Sep 16 17:52:08.976566 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:08.976492    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ad9133d-d3a8-420c-8ee2-ab7aff0c675d-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h\" (UID: \"3ad9133d-d3a8-420c-8ee2-ab7aff0c675d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h"
Sep 16 17:52:09.077012 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:09.076961    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ad9133d-d3a8-420c-8ee2-ab7aff0c675d-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h\" (UID: \"3ad9133d-d3a8-420c-8ee2-ab7aff0c675d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h"
Sep 16 17:52:09.077228 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:09.077050    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rkmj\" (UniqueName: \"kubernetes.io/projected/3ad9133d-d3a8-420c-8ee2-ab7aff0c675d-kube-api-access-4rkmj\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h\" (UID: \"3ad9133d-d3a8-420c-8ee2-ab7aff0c675d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h"
Sep 16 17:52:09.079774 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:09.079744    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ad9133d-d3a8-420c-8ee2-ab7aff0c675d-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h\" (UID: \"3ad9133d-d3a8-420c-8ee2-ab7aff0c675d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h"
Sep 16 17:52:09.087613 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:09.087592    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rkmj\" (UniqueName: \"kubernetes.io/projected/3ad9133d-d3a8-420c-8ee2-ab7aff0c675d-kube-api-access-4rkmj\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h\" (UID: \"3ad9133d-d3a8-420c-8ee2-ab7aff0c675d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h"
Sep 16 17:52:09.149903 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:09.149857    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6"]
Sep 16 17:52:09.152225 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:52:09.152193    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d9032ed_5be7_4881_a09f_309c9dc48f3d.slice/crio-3242f50e67d08669b640eccf81cae77ecc453fcbc793ba9835edcaa7950a9e56 WatchSource:0}: Error finding container 3242f50e67d08669b640eccf81cae77ecc453fcbc793ba9835edcaa7950a9e56: Status 404 returned error can't find the container with id 3242f50e67d08669b640eccf81cae77ecc453fcbc793ba9835edcaa7950a9e56
Sep 16 17:52:09.164627 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:09.164605    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h"
Sep 16 17:52:09.322239 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:09.322203    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6" event={"ID":"7d9032ed-5be7-4881-a09f-309c9dc48f3d","Type":"ContainerStarted","Data":"33ae2d6539f1cdc5d870b9b4b9a0701b0704b6cdf735b17830f1e39a1cc4d5b2"}
Sep 16 17:52:09.322452 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:09.322265    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6" event={"ID":"7d9032ed-5be7-4881-a09f-309c9dc48f3d","Type":"ContainerStarted","Data":"3242f50e67d08669b640eccf81cae77ecc453fcbc793ba9835edcaa7950a9e56"}
Sep 16 17:52:09.344221 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:09.344162    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6" podStartSLOduration=1.344143307 podStartE2EDuration="1.344143307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:52:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:52:09.343286679 +0000 UTC m=+1650.705965477" watchObservedRunningTime="2026-09-16 17:52:09.344143307 +0000 UTC m=+1650.706822094"
Sep 16 17:52:09.446500 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:09.446472    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h"]
Sep 16 17:52:09.448282 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:52:09.448255    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ad9133d_d3a8_420c_8ee2_ab7aff0c675d.slice/crio-a14d8c55ef8d498cb5cc954bccdd03be3d7f853a8237b38d28ba9c299fd7eb46 WatchSource:0}: Error finding container a14d8c55ef8d498cb5cc954bccdd03be3d7f853a8237b38d28ba9c299fd7eb46: Status 404 returned error can't find the container with id a14d8c55ef8d498cb5cc954bccdd03be3d7f853a8237b38d28ba9c299fd7eb46
Sep 16 17:52:09.869910 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:09.869835    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6"
Sep 16 17:52:10.330144 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:10.330104    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h" event={"ID":"3ad9133d-d3a8-420c-8ee2-ab7aff0c675d","Type":"ContainerStarted","Data":"add8a06421cc1ed20b6b7c41cd4ce7ef1f2296c23aba9ae233c8e921c8d55ce4"}
Sep 16 17:52:10.330316 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:10.330161    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h" event={"ID":"3ad9133d-d3a8-420c-8ee2-ab7aff0c675d","Type":"ContainerStarted","Data":"a14d8c55ef8d498cb5cc954bccdd03be3d7f853a8237b38d28ba9c299fd7eb46"}
Sep 16 17:52:10.351004 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:10.350948    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h" podStartSLOduration=2.350931626 podStartE2EDuration="2.350931626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:52:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:52:10.348637771 +0000 UTC m=+1651.711316558" watchObservedRunningTime="2026-09-16 17:52:10.350931626 +0000 UTC m=+1651.713610411"
Sep 16 17:52:11.336168 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:11.336127    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h"
Sep 16 17:52:14.118103 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:14.118063    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl" podUID="525daaaf-c2af-4bf8-bf27-d777c52a5147" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 16 17:52:18.869068 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:18.869028    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6"
Sep 16 17:52:18.881892 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:18.881835    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6"
Sep 16 17:52:19.399180 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:19.399146    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6"
Sep 16 17:52:23.650145 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:23.650100    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:52:24.117722 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:24.117677    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl" podUID="525daaaf-c2af-4bf8-bf27-d777c52a5147" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 16 17:52:28.696577 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:28.696536    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:52:34.118476 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:34.118433    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl" podUID="525daaaf-c2af-4bf8-bf27-d777c52a5147" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 16 17:52:42.343088 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:42.343057    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h"
Sep 16 17:52:43.934177 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:43.934136    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:52:43.942879 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:43.942842    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h"]
Sep 16 17:52:43.951131 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:43.951100    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6"]
Sep 16 17:52:43.960215 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:52:43.960176    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 17:52:43.960215 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:52:43.960215 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 17:52:43.960215 ip-10-0-136-122 kubenswrapper[2577]:  > containerID="33ae2d6539f1cdc5d870b9b4b9a0701b0704b6cdf735b17830f1e39a1cc4d5b2" cmd=["/bin/sleep","15"]
Sep 16 17:52:43.960471 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:43.960236    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6" message="" err=<
Sep 16 17:52:43.960471 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:52:43.960471 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 17:52:43.960471 ip-10-0-136-122 kubenswrapper[2577]:  >
Sep 16 17:52:43.960471 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:52:43.960254    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 17:52:43.960471 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:52:43.960471 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 17:52:43.960471 ip-10-0-136-122 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6" podUID="7d9032ed-5be7-4881-a09f-309c9dc48f3d" containerName="main" containerID="cri-o://33ae2d6539f1cdc5d870b9b4b9a0701b0704b6cdf735b17830f1e39a1cc4d5b2"
Sep 16 17:52:43.960471 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:43.960325    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6" podUID="7d9032ed-5be7-4881-a09f-309c9dc48f3d" containerName="main" containerID="cri-o://33ae2d6539f1cdc5d870b9b4b9a0701b0704b6cdf735b17830f1e39a1cc4d5b2" gracePeriod=60
Sep 16 17:52:44.027891 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:44.027832    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:52:44.118775 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:44.118732    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl" podUID="525daaaf-c2af-4bf8-bf27-d777c52a5147" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 16 17:52:44.320077 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:44.320051    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6"
Sep 16 17:52:44.395220 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:44.395178    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9032ed-5be7-4881-a09f-309c9dc48f3d-tls-certs\") pod \"7d9032ed-5be7-4881-a09f-309c9dc48f3d\" (UID: \"7d9032ed-5be7-4881-a09f-309c9dc48f3d\") "
Sep 16 17:52:44.395220 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:44.395221    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9032ed-5be7-4881-a09f-309c9dc48f3d-dshm\") pod \"7d9032ed-5be7-4881-a09f-309c9dc48f3d\" (UID: \"7d9032ed-5be7-4881-a09f-309c9dc48f3d\") "
Sep 16 17:52:44.395469 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:44.395286    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9032ed-5be7-4881-a09f-309c9dc48f3d-tmp-dir\") pod \"7d9032ed-5be7-4881-a09f-309c9dc48f3d\" (UID: \"7d9032ed-5be7-4881-a09f-309c9dc48f3d\") "
Sep 16 17:52:44.395469 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:44.395337    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9032ed-5be7-4881-a09f-309c9dc48f3d-model-cache\") pod \"7d9032ed-5be7-4881-a09f-309c9dc48f3d\" (UID: \"7d9032ed-5be7-4881-a09f-309c9dc48f3d\") "
Sep 16 17:52:44.395469 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:44.395369    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d9032ed-5be7-4881-a09f-309c9dc48f3d-home\") pod \"7d9032ed-5be7-4881-a09f-309c9dc48f3d\" (UID: \"7d9032ed-5be7-4881-a09f-309c9dc48f3d\") "
Sep 16 17:52:44.395469 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:44.395409    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ss2jj\" (UniqueName: \"kubernetes.io/projected/7d9032ed-5be7-4881-a09f-309c9dc48f3d-kube-api-access-ss2jj\") pod \"7d9032ed-5be7-4881-a09f-309c9dc48f3d\" (UID: \"7d9032ed-5be7-4881-a09f-309c9dc48f3d\") "
Sep 16 17:52:44.395681 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:44.395568    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9032ed-5be7-4881-a09f-309c9dc48f3d-model-cache" (OuterVolumeSpecName: "model-cache") pod "7d9032ed-5be7-4881-a09f-309c9dc48f3d" (UID: "7d9032ed-5be7-4881-a09f-309c9dc48f3d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:52:44.395681 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:44.395588    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9032ed-5be7-4881-a09f-309c9dc48f3d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7d9032ed-5be7-4881-a09f-309c9dc48f3d" (UID: "7d9032ed-5be7-4881-a09f-309c9dc48f3d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:52:44.395778 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:44.395716    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9032ed-5be7-4881-a09f-309c9dc48f3d-home" (OuterVolumeSpecName: "home") pod "7d9032ed-5be7-4881-a09f-309c9dc48f3d" (UID: "7d9032ed-5be7-4881-a09f-309c9dc48f3d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:52:44.395834 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:44.395782    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9032ed-5be7-4881-a09f-309c9dc48f3d-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:52:44.395834 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:44.395804    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9032ed-5be7-4881-a09f-309c9dc48f3d-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:52:44.397590 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:44.397554    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d9032ed-5be7-4881-a09f-309c9dc48f3d-kube-api-access-ss2jj" (OuterVolumeSpecName: "kube-api-access-ss2jj") pod "7d9032ed-5be7-4881-a09f-309c9dc48f3d" (UID: "7d9032ed-5be7-4881-a09f-309c9dc48f3d"). InnerVolumeSpecName "kube-api-access-ss2jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:52:44.397590 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:44.397579    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d9032ed-5be7-4881-a09f-309c9dc48f3d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7d9032ed-5be7-4881-a09f-309c9dc48f3d" (UID: "7d9032ed-5be7-4881-a09f-309c9dc48f3d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:52:44.397802 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:44.397604    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9032ed-5be7-4881-a09f-309c9dc48f3d-dshm" (OuterVolumeSpecName: "dshm") pod "7d9032ed-5be7-4881-a09f-309c9dc48f3d" (UID: "7d9032ed-5be7-4881-a09f-309c9dc48f3d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:52:44.496660 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:44.496561    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d9032ed-5be7-4881-a09f-309c9dc48f3d-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:52:44.496660 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:44.496595    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss2jj\" (UniqueName: \"kubernetes.io/projected/7d9032ed-5be7-4881-a09f-309c9dc48f3d-kube-api-access-ss2jj\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:52:44.496660 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:44.496607    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9032ed-5be7-4881-a09f-309c9dc48f3d-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:52:44.496660 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:44.496621    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9032ed-5be7-4881-a09f-309c9dc48f3d-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:52:44.543962 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:44.543925    2577 generic.go:356] "Generic (PLEG): container finished" podID="7d9032ed-5be7-4881-a09f-309c9dc48f3d" containerID="33ae2d6539f1cdc5d870b9b4b9a0701b0704b6cdf735b17830f1e39a1cc4d5b2" exitCode=0
Sep 16 17:52:44.544161 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:44.543960    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6" event={"ID":"7d9032ed-5be7-4881-a09f-309c9dc48f3d","Type":"ContainerDied","Data":"33ae2d6539f1cdc5d870b9b4b9a0701b0704b6cdf735b17830f1e39a1cc4d5b2"}
Sep 16 17:52:44.544161 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:44.543999    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6"
Sep 16 17:52:44.544161 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:44.544034    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6" event={"ID":"7d9032ed-5be7-4881-a09f-309c9dc48f3d","Type":"ContainerDied","Data":"3242f50e67d08669b640eccf81cae77ecc453fcbc793ba9835edcaa7950a9e56"}
Sep 16 17:52:44.544161 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:44.544068    2577 scope.go:117] "RemoveContainer" containerID="33ae2d6539f1cdc5d870b9b4b9a0701b0704b6cdf735b17830f1e39a1cc4d5b2"
Sep 16 17:52:44.554061 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:44.554039    2577 scope.go:117] "RemoveContainer" containerID="33ae2d6539f1cdc5d870b9b4b9a0701b0704b6cdf735b17830f1e39a1cc4d5b2"
Sep 16 17:52:44.554413 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:52:44.554392    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33ae2d6539f1cdc5d870b9b4b9a0701b0704b6cdf735b17830f1e39a1cc4d5b2\": container with ID starting with 33ae2d6539f1cdc5d870b9b4b9a0701b0704b6cdf735b17830f1e39a1cc4d5b2 not found: ID does not exist" containerID="33ae2d6539f1cdc5d870b9b4b9a0701b0704b6cdf735b17830f1e39a1cc4d5b2"
Sep 16 17:52:44.554480 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:44.554427    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33ae2d6539f1cdc5d870b9b4b9a0701b0704b6cdf735b17830f1e39a1cc4d5b2"} err="failed to get container status \"33ae2d6539f1cdc5d870b9b4b9a0701b0704b6cdf735b17830f1e39a1cc4d5b2\": rpc error: code = NotFound desc = could not find container \"33ae2d6539f1cdc5d870b9b4b9a0701b0704b6cdf735b17830f1e39a1cc4d5b2\": container with ID starting with 33ae2d6539f1cdc5d870b9b4b9a0701b0704b6cdf735b17830f1e39a1cc4d5b2 not found: ID does not exist"
Sep 16 17:52:44.571097 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:44.571069    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6"]
Sep 16 17:52:44.575413 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:44.575390    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-mlst6"]
Sep 16 17:52:45.084462 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:45.084425    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d9032ed-5be7-4881-a09f-309c9dc48f3d" path="/var/lib/kubelet/pods/7d9032ed-5be7-4881-a09f-309c9dc48f3d/volumes"
Sep 16 17:52:52.343279 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:52.343241    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h" podUID="3ad9133d-d3a8-420c-8ee2-ab7aff0c675d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:52:54.117846 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:54.117791    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl" podUID="525daaaf-c2af-4bf8-bf27-d777c52a5147" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 16 17:52:55.762887 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:55.762831    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h" message="" err="command '/bin/sleep 15' exited with 137: "
Sep 16 17:52:55.762887 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:52:55.762884    2577 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h" podUID="3ad9133d-d3a8-420c-8ee2-ab7aff0c675d" containerName="main" containerID="cri-o://add8a06421cc1ed20b6b7c41cd4ce7ef1f2296c23aba9ae233c8e921c8d55ce4"
Sep 16 17:52:55.763286 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:55.762938    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h" podUID="3ad9133d-d3a8-420c-8ee2-ab7aff0c675d" containerName="main" containerID="cri-o://add8a06421cc1ed20b6b7c41cd4ce7ef1f2296c23aba9ae233c8e921c8d55ce4" gracePeriod=49
Sep 16 17:52:55.926019 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:55.925994    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h_3ad9133d-d3a8-420c-8ee2-ab7aff0c675d/main/0.log"
Sep 16 17:52:55.928330 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:55.928314    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h"
Sep 16 17:52:55.995264 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:55.995226    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ad9133d-d3a8-420c-8ee2-ab7aff0c675d-tls-certs\") pod \"3ad9133d-d3a8-420c-8ee2-ab7aff0c675d\" (UID: \"3ad9133d-d3a8-420c-8ee2-ab7aff0c675d\") "
Sep 16 17:52:55.995427 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:55.995376    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rkmj\" (UniqueName: \"kubernetes.io/projected/3ad9133d-d3a8-420c-8ee2-ab7aff0c675d-kube-api-access-4rkmj\") pod \"3ad9133d-d3a8-420c-8ee2-ab7aff0c675d\" (UID: \"3ad9133d-d3a8-420c-8ee2-ab7aff0c675d\") "
Sep 16 17:52:55.997222 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:55.997184    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ad9133d-d3a8-420c-8ee2-ab7aff0c675d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3ad9133d-d3a8-420c-8ee2-ab7aff0c675d" (UID: "3ad9133d-d3a8-420c-8ee2-ab7aff0c675d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:52:55.997335 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:55.997309    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ad9133d-d3a8-420c-8ee2-ab7aff0c675d-kube-api-access-4rkmj" (OuterVolumeSpecName: "kube-api-access-4rkmj") pod "3ad9133d-d3a8-420c-8ee2-ab7aff0c675d" (UID: "3ad9133d-d3a8-420c-8ee2-ab7aff0c675d"). InnerVolumeSpecName "kube-api-access-4rkmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:52:56.096034 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:56.096002    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ad9133d-d3a8-420c-8ee2-ab7aff0c675d-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:52:56.096034 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:56.096028    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rkmj\" (UniqueName: \"kubernetes.io/projected/3ad9133d-d3a8-420c-8ee2-ab7aff0c675d-kube-api-access-4rkmj\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:52:56.616112 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:56.616085    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h_3ad9133d-d3a8-420c-8ee2-ab7aff0c675d/main/0.log"
Sep 16 17:52:56.616282 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:56.616125    2577 generic.go:356] "Generic (PLEG): container finished" podID="3ad9133d-d3a8-420c-8ee2-ab7aff0c675d" containerID="add8a06421cc1ed20b6b7c41cd4ce7ef1f2296c23aba9ae233c8e921c8d55ce4" exitCode=1
Sep 16 17:52:56.616282 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:56.616159    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h" event={"ID":"3ad9133d-d3a8-420c-8ee2-ab7aff0c675d","Type":"ContainerDied","Data":"add8a06421cc1ed20b6b7c41cd4ce7ef1f2296c23aba9ae233c8e921c8d55ce4"}
Sep 16 17:52:56.616282 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:56.616193    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h"
Sep 16 17:52:56.616282 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:56.616227    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h" event={"ID":"3ad9133d-d3a8-420c-8ee2-ab7aff0c675d","Type":"ContainerDied","Data":"a14d8c55ef8d498cb5cc954bccdd03be3d7f853a8237b38d28ba9c299fd7eb46"}
Sep 16 17:52:56.616282 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:56.616251    2577 scope.go:117] "RemoveContainer" containerID="add8a06421cc1ed20b6b7c41cd4ce7ef1f2296c23aba9ae233c8e921c8d55ce4"
Sep 16 17:52:56.625255 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:56.625234    2577 scope.go:117] "RemoveContainer" containerID="add8a06421cc1ed20b6b7c41cd4ce7ef1f2296c23aba9ae233c8e921c8d55ce4"
Sep 16 17:52:56.625518 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:52:56.625500    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"add8a06421cc1ed20b6b7c41cd4ce7ef1f2296c23aba9ae233c8e921c8d55ce4\": container with ID starting with add8a06421cc1ed20b6b7c41cd4ce7ef1f2296c23aba9ae233c8e921c8d55ce4 not found: ID does not exist" containerID="add8a06421cc1ed20b6b7c41cd4ce7ef1f2296c23aba9ae233c8e921c8d55ce4"
Sep 16 17:52:56.625587 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:56.625527    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"add8a06421cc1ed20b6b7c41cd4ce7ef1f2296c23aba9ae233c8e921c8d55ce4"} err="failed to get container status \"add8a06421cc1ed20b6b7c41cd4ce7ef1f2296c23aba9ae233c8e921c8d55ce4\": rpc error: code = NotFound desc = could not find container \"add8a06421cc1ed20b6b7c41cd4ce7ef1f2296c23aba9ae233c8e921c8d55ce4\": container with ID starting with add8a06421cc1ed20b6b7c41cd4ce7ef1f2296c23aba9ae233c8e921c8d55ce4 not found: ID does not exist"
Sep 16 17:52:56.642317 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:56.642286    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h"]
Sep 16 17:52:56.648585 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:56.648555    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc52kk9h"]
Sep 16 17:52:57.080367 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:57.080339    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ad9133d-d3a8-420c-8ee2-ab7aff0c675d" path="/var/lib/kubelet/pods/3ad9133d-d3a8-420c-8ee2-ab7aff0c675d/volumes"
Sep 16 17:52:57.349591 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:52:57.349503    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:53:04.118616 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:04.118525    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl" podUID="525daaaf-c2af-4bf8-bf27-d777c52a5147" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 16 17:53:14.118110 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:14.118059    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl" podUID="525daaaf-c2af-4bf8-bf27-d777c52a5147" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 16 17:53:21.081752 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:21.081709    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:53:24.118047 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:24.117995    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl" podUID="525daaaf-c2af-4bf8-bf27-d777c52a5147" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 16 17:53:32.439511 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:32.439454    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:53:33.667182 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:33.667130    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:53:33.826576 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:33.826529    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:53:34.118276 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:34.118236    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl" podUID="525daaaf-c2af-4bf8-bf27-d777c52a5147" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 16 17:53:36.108504 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.108469    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk"]
Sep 16 17:53:36.108918 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.108826    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ad9133d-d3a8-420c-8ee2-ab7aff0c675d" containerName="main"
Sep 16 17:53:36.108918 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.108838    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ad9133d-d3a8-420c-8ee2-ab7aff0c675d" containerName="main"
Sep 16 17:53:36.108918 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.108857    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d9032ed-5be7-4881-a09f-309c9dc48f3d" containerName="main"
Sep 16 17:53:36.108918 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.108879    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d9032ed-5be7-4881-a09f-309c9dc48f3d" containerName="main"
Sep 16 17:53:36.109064 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.108955    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ad9133d-d3a8-420c-8ee2-ab7aff0c675d" containerName="main"
Sep 16 17:53:36.109064 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.108964    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d9032ed-5be7-4881-a09f-309c9dc48f3d" containerName="main"
Sep 16 17:53:36.113348 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.113327    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk"
Sep 16 17:53:36.119899 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.119874    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-smzct\""
Sep 16 17:53:36.120046 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.119913    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 16 17:53:36.120046 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.120001    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 16 17:53:36.120499 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.119920    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 16 17:53:36.151849 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.145522    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c99bc\" (UniqueName: \"kubernetes.io/projected/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-kube-api-access-c99bc\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hxrhk\" (UID: \"4fa789b8-bd57-46b8-84a1-ac378bd5a32e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk"
Sep 16 17:53:36.151849 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.145593    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hxrhk\" (UID: \"4fa789b8-bd57-46b8-84a1-ac378bd5a32e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk"
Sep 16 17:53:36.151849 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.145631    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hxrhk\" (UID: \"4fa789b8-bd57-46b8-84a1-ac378bd5a32e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk"
Sep 16 17:53:36.151849 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.145699    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hxrhk\" (UID: \"4fa789b8-bd57-46b8-84a1-ac378bd5a32e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk"
Sep 16 17:53:36.151849 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.145739    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hxrhk\" (UID: \"4fa789b8-bd57-46b8-84a1-ac378bd5a32e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk"
Sep 16 17:53:36.151849 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.145776    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hxrhk\" (UID: \"4fa789b8-bd57-46b8-84a1-ac378bd5a32e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk"
Sep 16 17:53:36.156004 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.155969    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk"]
Sep 16 17:53:36.168214 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.168182    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:53:36.246695 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.246656    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hxrhk\" (UID: \"4fa789b8-bd57-46b8-84a1-ac378bd5a32e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk"
Sep 16 17:53:36.246903 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.246715    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c99bc\" (UniqueName: \"kubernetes.io/projected/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-kube-api-access-c99bc\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hxrhk\" (UID: \"4fa789b8-bd57-46b8-84a1-ac378bd5a32e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk"
Sep 16 17:53:36.246903 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.246752    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hxrhk\" (UID: \"4fa789b8-bd57-46b8-84a1-ac378bd5a32e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk"
Sep 16 17:53:36.246903 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.246778    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hxrhk\" (UID: \"4fa789b8-bd57-46b8-84a1-ac378bd5a32e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk"
Sep 16 17:53:36.246903 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.246821    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hxrhk\" (UID: \"4fa789b8-bd57-46b8-84a1-ac378bd5a32e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk"
Sep 16 17:53:36.246903 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.246851    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hxrhk\" (UID: \"4fa789b8-bd57-46b8-84a1-ac378bd5a32e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk"
Sep 16 17:53:36.247199 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.247171    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hxrhk\" (UID: \"4fa789b8-bd57-46b8-84a1-ac378bd5a32e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk"
Sep 16 17:53:36.247199 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.247186    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hxrhk\" (UID: \"4fa789b8-bd57-46b8-84a1-ac378bd5a32e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk"
Sep 16 17:53:36.247305 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.247238    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hxrhk\" (UID: \"4fa789b8-bd57-46b8-84a1-ac378bd5a32e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk"
Sep 16 17:53:36.249140 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.249113    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hxrhk\" (UID: \"4fa789b8-bd57-46b8-84a1-ac378bd5a32e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk"
Sep 16 17:53:36.249409 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.249391    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hxrhk\" (UID: \"4fa789b8-bd57-46b8-84a1-ac378bd5a32e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk"
Sep 16 17:53:36.257450 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.257425    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c99bc\" (UniqueName: \"kubernetes.io/projected/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-kube-api-access-c99bc\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hxrhk\" (UID: \"4fa789b8-bd57-46b8-84a1-ac378bd5a32e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk"
Sep 16 17:53:36.424008 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.423917    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk"
Sep 16 17:53:36.484245 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.484210    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:53:36.733191 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.733157    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk"]
Sep 16 17:53:36.733191 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:53:36.733173    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa789b8_bd57_46b8_84a1_ac378bd5a32e.slice/crio-f7ce8147a30e42acdc9f857fa08b02eb839e09c9f7c4a4ff17aec2b8ef574c41 WatchSource:0}: Error finding container f7ce8147a30e42acdc9f857fa08b02eb839e09c9f7c4a4ff17aec2b8ef574c41: Status 404 returned error can't find the container with id f7ce8147a30e42acdc9f857fa08b02eb839e09c9f7c4a4ff17aec2b8ef574c41
Sep 16 17:53:36.856459 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.856422    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk" event={"ID":"4fa789b8-bd57-46b8-84a1-ac378bd5a32e","Type":"ContainerStarted","Data":"20d800c65c813ae7051a1a5159db43b2551cf176785ebeb7884be52d320bc3d1"}
Sep 16 17:53:36.856626 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.856488    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk" event={"ID":"4fa789b8-bd57-46b8-84a1-ac378bd5a32e","Type":"ContainerStarted","Data":"f7ce8147a30e42acdc9f857fa08b02eb839e09c9f7c4a4ff17aec2b8ef574c41"}
Sep 16 17:53:36.891133 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:36.891083    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk" podStartSLOduration=0.891066354 podStartE2EDuration="891.066354ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:53:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:53:36.890838555 +0000 UTC m=+1738.253517341" watchObservedRunningTime="2026-09-16 17:53:36.891066354 +0000 UTC m=+1738.253745140"
Sep 16 17:53:37.424594 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:37.424561    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk"
Sep 16 17:53:41.475804 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:41.475745    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:53:43.157298 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:43.157263    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:53:44.118458 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:44.118415    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl" podUID="525daaaf-c2af-4bf8-bf27-d777c52a5147" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 16 17:53:46.424491 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:46.424458    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk"
Sep 16 17:53:46.444433 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:46.444404    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk"
Sep 16 17:53:46.926779 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:46.926750    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk"
Sep 16 17:53:48.066149 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:48.066116    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:53:48.164292 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:48.164240    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:53:54.127653 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:54.127619    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"
Sep 16 17:53:54.135424 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:54.135398    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"
Sep 16 17:53:58.822993 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:58.822942    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:53:58.833294 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:58.833265    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"]
Sep 16 17:53:58.923009 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:53:58.922971    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:54:10.729960 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:10.729916    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:54:10.737501 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:10.737466    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk"]
Sep 16 17:54:10.747170 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:54:10.747126    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 17:54:10.747170 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:54:10.747170 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 17:54:10.747170 ip-10-0-136-122 kubenswrapper[2577]:  > containerID="20d800c65c813ae7051a1a5159db43b2551cf176785ebeb7884be52d320bc3d1" cmd=["/bin/sleep","15"]
Sep 16 17:54:10.747508 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:10.747190    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk" message="" err=<
Sep 16 17:54:10.747508 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:54:10.747508 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 17:54:10.747508 ip-10-0-136-122 kubenswrapper[2577]:  >
Sep 16 17:54:10.747508 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:54:10.747202    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 17:54:10.747508 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:54:10.747508 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 17:54:10.747508 ip-10-0-136-122 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk" podUID="4fa789b8-bd57-46b8-84a1-ac378bd5a32e" containerName="main" containerID="cri-o://20d800c65c813ae7051a1a5159db43b2551cf176785ebeb7884be52d320bc3d1"
Sep 16 17:54:10.747508 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:10.747248    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk" podUID="4fa789b8-bd57-46b8-84a1-ac378bd5a32e" containerName="main" containerID="cri-o://20d800c65c813ae7051a1a5159db43b2551cf176785ebeb7884be52d320bc3d1" gracePeriod=60
Sep 16 17:54:10.845155 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:10.845123    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:54:10.916157 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:10.916115    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk" podUID="4fa789b8-bd57-46b8-84a1-ac378bd5a32e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 16 17:54:11.049857 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:11.049772    2577 generic.go:356] "Generic (PLEG): container finished" podID="4fa789b8-bd57-46b8-84a1-ac378bd5a32e" containerID="20d800c65c813ae7051a1a5159db43b2551cf176785ebeb7884be52d320bc3d1" exitCode=0
Sep 16 17:54:11.049857 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:11.049834    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk" event={"ID":"4fa789b8-bd57-46b8-84a1-ac378bd5a32e","Type":"ContainerDied","Data":"20d800c65c813ae7051a1a5159db43b2551cf176785ebeb7884be52d320bc3d1"}
Sep 16 17:54:11.505389 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:11.505363    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk"
Sep 16 17:54:11.671462 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:11.671375    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-home\") pod \"4fa789b8-bd57-46b8-84a1-ac378bd5a32e\" (UID: \"4fa789b8-bd57-46b8-84a1-ac378bd5a32e\") "
Sep 16 17:54:11.671462 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:11.671414    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-model-cache\") pod \"4fa789b8-bd57-46b8-84a1-ac378bd5a32e\" (UID: \"4fa789b8-bd57-46b8-84a1-ac378bd5a32e\") "
Sep 16 17:54:11.671462 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:11.671439    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-dshm\") pod \"4fa789b8-bd57-46b8-84a1-ac378bd5a32e\" (UID: \"4fa789b8-bd57-46b8-84a1-ac378bd5a32e\") "
Sep 16 17:54:11.671767 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:11.671470    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-tls-certs\") pod \"4fa789b8-bd57-46b8-84a1-ac378bd5a32e\" (UID: \"4fa789b8-bd57-46b8-84a1-ac378bd5a32e\") "
Sep 16 17:54:11.671767 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:11.671551    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-tmp-dir\") pod \"4fa789b8-bd57-46b8-84a1-ac378bd5a32e\" (UID: \"4fa789b8-bd57-46b8-84a1-ac378bd5a32e\") "
Sep 16 17:54:11.671767 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:11.671584    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c99bc\" (UniqueName: \"kubernetes.io/projected/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-kube-api-access-c99bc\") pod \"4fa789b8-bd57-46b8-84a1-ac378bd5a32e\" (UID: \"4fa789b8-bd57-46b8-84a1-ac378bd5a32e\") "
Sep 16 17:54:11.671767 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:11.671710    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-home" (OuterVolumeSpecName: "home") pod "4fa789b8-bd57-46b8-84a1-ac378bd5a32e" (UID: "4fa789b8-bd57-46b8-84a1-ac378bd5a32e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:54:11.671767 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:11.671728    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-model-cache" (OuterVolumeSpecName: "model-cache") pod "4fa789b8-bd57-46b8-84a1-ac378bd5a32e" (UID: "4fa789b8-bd57-46b8-84a1-ac378bd5a32e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:54:11.672037 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:11.671826    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4fa789b8-bd57-46b8-84a1-ac378bd5a32e" (UID: "4fa789b8-bd57-46b8-84a1-ac378bd5a32e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:54:11.672037 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:11.671926    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:54:11.672037 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:11.671948    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:54:11.672037 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:11.671965    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:54:11.673693 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:11.673663    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-dshm" (OuterVolumeSpecName: "dshm") pod "4fa789b8-bd57-46b8-84a1-ac378bd5a32e" (UID: "4fa789b8-bd57-46b8-84a1-ac378bd5a32e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:54:11.673821 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:11.673762    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4fa789b8-bd57-46b8-84a1-ac378bd5a32e" (UID: "4fa789b8-bd57-46b8-84a1-ac378bd5a32e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:54:11.673821 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:11.673776    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-kube-api-access-c99bc" (OuterVolumeSpecName: "kube-api-access-c99bc") pod "4fa789b8-bd57-46b8-84a1-ac378bd5a32e" (UID: "4fa789b8-bd57-46b8-84a1-ac378bd5a32e"). InnerVolumeSpecName "kube-api-access-c99bc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:54:11.773243 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:11.773191    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c99bc\" (UniqueName: \"kubernetes.io/projected/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-kube-api-access-c99bc\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:54:11.773243 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:11.773237    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:54:11.773243 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:11.773248    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fa789b8-bd57-46b8-84a1-ac378bd5a32e-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:54:12.057204 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:12.057115    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk"
Sep 16 17:54:12.057204 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:12.057166    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk" event={"ID":"4fa789b8-bd57-46b8-84a1-ac378bd5a32e","Type":"ContainerDied","Data":"f7ce8147a30e42acdc9f857fa08b02eb839e09c9f7c4a4ff17aec2b8ef574c41"}
Sep 16 17:54:12.057432 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:12.057235    2577 scope.go:117] "RemoveContainer" containerID="20d800c65c813ae7051a1a5159db43b2551cf176785ebeb7884be52d320bc3d1"
Sep 16 17:54:12.082660 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:12.082623    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk"]
Sep 16 17:54:12.086120 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:12.086097    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hxrhk"]
Sep 16 17:54:13.080712 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:13.080683    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fa789b8-bd57-46b8-84a1-ac378bd5a32e" path="/var/lib/kubelet/pods/4fa789b8-bd57-46b8-84a1-ac378bd5a32e/volumes"
Sep 16 17:54:13.855741 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:13.855701    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl" podUID="525daaaf-c2af-4bf8-bf27-d777c52a5147" containerName="main" containerID="cri-o://3b622bb0ffb1b8265e6e8afec5c1be40c8958e8ff07b12d4adebb6a440a42d89" gracePeriod=45
Sep 16 17:54:14.128733 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:14.128640    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl" podUID="525daaaf-c2af-4bf8-bf27-d777c52a5147" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 16 17:54:14.602690 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:14.602665    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"
Sep 16 17:54:14.798891 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:14.798786    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/525daaaf-c2af-4bf8-bf27-d777c52a5147-tls-certs\") pod \"525daaaf-c2af-4bf8-bf27-d777c52a5147\" (UID: \"525daaaf-c2af-4bf8-bf27-d777c52a5147\") "
Sep 16 17:54:14.798891 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:14.798832    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/525daaaf-c2af-4bf8-bf27-d777c52a5147-model-cache\") pod \"525daaaf-c2af-4bf8-bf27-d777c52a5147\" (UID: \"525daaaf-c2af-4bf8-bf27-d777c52a5147\") "
Sep 16 17:54:14.798891 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:14.798889    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/525daaaf-c2af-4bf8-bf27-d777c52a5147-dshm\") pod \"525daaaf-c2af-4bf8-bf27-d777c52a5147\" (UID: \"525daaaf-c2af-4bf8-bf27-d777c52a5147\") "
Sep 16 17:54:14.799212 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:14.799047    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/525daaaf-c2af-4bf8-bf27-d777c52a5147-home\") pod \"525daaaf-c2af-4bf8-bf27-d777c52a5147\" (UID: \"525daaaf-c2af-4bf8-bf27-d777c52a5147\") "
Sep 16 17:54:14.799212 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:14.799081    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525daaaf-c2af-4bf8-bf27-d777c52a5147-model-cache" (OuterVolumeSpecName: "model-cache") pod "525daaaf-c2af-4bf8-bf27-d777c52a5147" (UID: "525daaaf-c2af-4bf8-bf27-d777c52a5147"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:54:14.799212 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:14.799096    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/525daaaf-c2af-4bf8-bf27-d777c52a5147-kserve-provision-location\") pod \"525daaaf-c2af-4bf8-bf27-d777c52a5147\" (UID: \"525daaaf-c2af-4bf8-bf27-d777c52a5147\") "
Sep 16 17:54:14.799212 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:14.799129    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dk8f\" (UniqueName: \"kubernetes.io/projected/525daaaf-c2af-4bf8-bf27-d777c52a5147-kube-api-access-9dk8f\") pod \"525daaaf-c2af-4bf8-bf27-d777c52a5147\" (UID: \"525daaaf-c2af-4bf8-bf27-d777c52a5147\") "
Sep 16 17:54:14.799212 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:14.799178    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/525daaaf-c2af-4bf8-bf27-d777c52a5147-tmp-dir\") pod \"525daaaf-c2af-4bf8-bf27-d777c52a5147\" (UID: \"525daaaf-c2af-4bf8-bf27-d777c52a5147\") "
Sep 16 17:54:14.799212 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:14.799208    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/525daaaf-c2af-4bf8-bf27-d777c52a5147-cabundle-cert\") pod \"525daaaf-c2af-4bf8-bf27-d777c52a5147\" (UID: \"525daaaf-c2af-4bf8-bf27-d777c52a5147\") "
Sep 16 17:54:14.799533 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:14.799467    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/525daaaf-c2af-4bf8-bf27-d777c52a5147-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:54:14.799816 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:14.799790    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/525daaaf-c2af-4bf8-bf27-d777c52a5147-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "525daaaf-c2af-4bf8-bf27-d777c52a5147" (UID: "525daaaf-c2af-4bf8-bf27-d777c52a5147"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:54:14.799927 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:14.799820    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525daaaf-c2af-4bf8-bf27-d777c52a5147-home" (OuterVolumeSpecName: "home") pod "525daaaf-c2af-4bf8-bf27-d777c52a5147" (UID: "525daaaf-c2af-4bf8-bf27-d777c52a5147"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:54:14.801515 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:14.801485    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/525daaaf-c2af-4bf8-bf27-d777c52a5147-kube-api-access-9dk8f" (OuterVolumeSpecName: "kube-api-access-9dk8f") pod "525daaaf-c2af-4bf8-bf27-d777c52a5147" (UID: "525daaaf-c2af-4bf8-bf27-d777c52a5147"). InnerVolumeSpecName "kube-api-access-9dk8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:54:14.801655 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:14.801635    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/525daaaf-c2af-4bf8-bf27-d777c52a5147-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "525daaaf-c2af-4bf8-bf27-d777c52a5147" (UID: "525daaaf-c2af-4bf8-bf27-d777c52a5147"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:54:14.801941 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:14.801915    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525daaaf-c2af-4bf8-bf27-d777c52a5147-dshm" (OuterVolumeSpecName: "dshm") pod "525daaaf-c2af-4bf8-bf27-d777c52a5147" (UID: "525daaaf-c2af-4bf8-bf27-d777c52a5147"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:54:14.812119 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:14.812094    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525daaaf-c2af-4bf8-bf27-d777c52a5147-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "525daaaf-c2af-4bf8-bf27-d777c52a5147" (UID: "525daaaf-c2af-4bf8-bf27-d777c52a5147"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:54:14.822564 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:14.822540    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/525daaaf-c2af-4bf8-bf27-d777c52a5147-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "525daaaf-c2af-4bf8-bf27-d777c52a5147" (UID: "525daaaf-c2af-4bf8-bf27-d777c52a5147"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:54:14.900880 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:14.900824    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dk8f\" (UniqueName: \"kubernetes.io/projected/525daaaf-c2af-4bf8-bf27-d777c52a5147-kube-api-access-9dk8f\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:54:14.901040 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:14.900897    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/525daaaf-c2af-4bf8-bf27-d777c52a5147-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:54:14.901040 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:14.900915    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/525daaaf-c2af-4bf8-bf27-d777c52a5147-cabundle-cert\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:54:14.901040 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:14.900926    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/525daaaf-c2af-4bf8-bf27-d777c52a5147-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:54:14.901040 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:14.900936    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/525daaaf-c2af-4bf8-bf27-d777c52a5147-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:54:14.901040 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:14.900947    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/525daaaf-c2af-4bf8-bf27-d777c52a5147-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:54:14.901040 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:14.900958    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/525daaaf-c2af-4bf8-bf27-d777c52a5147-kserve-provision-location\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:54:15.083369 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:15.083328    2577 generic.go:356] "Generic (PLEG): container finished" podID="525daaaf-c2af-4bf8-bf27-d777c52a5147" containerID="3b622bb0ffb1b8265e6e8afec5c1be40c8958e8ff07b12d4adebb6a440a42d89" exitCode=0
Sep 16 17:54:15.083547 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:15.083376    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"
Sep 16 17:54:15.083547 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:15.083403    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl" event={"ID":"525daaaf-c2af-4bf8-bf27-d777c52a5147","Type":"ContainerDied","Data":"3b622bb0ffb1b8265e6e8afec5c1be40c8958e8ff07b12d4adebb6a440a42d89"}
Sep 16 17:54:15.083547 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:15.083460    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl" event={"ID":"525daaaf-c2af-4bf8-bf27-d777c52a5147","Type":"ContainerDied","Data":"4490c65b6e81512a3ad995c325dec9d81988ab8a7c387df36ae87faaceb4cd51"}
Sep 16 17:54:15.083547 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:15.083485    2577 scope.go:117] "RemoveContainer" containerID="3b622bb0ffb1b8265e6e8afec5c1be40c8958e8ff07b12d4adebb6a440a42d89"
Sep 16 17:54:15.093204 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:15.093160    2577 scope.go:117] "RemoveContainer" containerID="291eaad44d9b316ab0780c3a4a51f856d4a6a17ef5d64bef7fe33dc7319ff37b"
Sep 16 17:54:15.101140 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:15.101123    2577 scope.go:117] "RemoveContainer" containerID="3b622bb0ffb1b8265e6e8afec5c1be40c8958e8ff07b12d4adebb6a440a42d89"
Sep 16 17:54:15.101371 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:54:15.101353    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b622bb0ffb1b8265e6e8afec5c1be40c8958e8ff07b12d4adebb6a440a42d89\": container with ID starting with 3b622bb0ffb1b8265e6e8afec5c1be40c8958e8ff07b12d4adebb6a440a42d89 not found: ID does not exist" containerID="3b622bb0ffb1b8265e6e8afec5c1be40c8958e8ff07b12d4adebb6a440a42d89"
Sep 16 17:54:15.101419 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:15.101378    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b622bb0ffb1b8265e6e8afec5c1be40c8958e8ff07b12d4adebb6a440a42d89"} err="failed to get container status \"3b622bb0ffb1b8265e6e8afec5c1be40c8958e8ff07b12d4adebb6a440a42d89\": rpc error: code = NotFound desc = could not find container \"3b622bb0ffb1b8265e6e8afec5c1be40c8958e8ff07b12d4adebb6a440a42d89\": container with ID starting with 3b622bb0ffb1b8265e6e8afec5c1be40c8958e8ff07b12d4adebb6a440a42d89 not found: ID does not exist"
Sep 16 17:54:15.101419 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:15.101398    2577 scope.go:117] "RemoveContainer" containerID="291eaad44d9b316ab0780c3a4a51f856d4a6a17ef5d64bef7fe33dc7319ff37b"
Sep 16 17:54:15.101651 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:54:15.101630    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"291eaad44d9b316ab0780c3a4a51f856d4a6a17ef5d64bef7fe33dc7319ff37b\": container with ID starting with 291eaad44d9b316ab0780c3a4a51f856d4a6a17ef5d64bef7fe33dc7319ff37b not found: ID does not exist" containerID="291eaad44d9b316ab0780c3a4a51f856d4a6a17ef5d64bef7fe33dc7319ff37b"
Sep 16 17:54:15.101689 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:15.101659    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"291eaad44d9b316ab0780c3a4a51f856d4a6a17ef5d64bef7fe33dc7319ff37b"} err="failed to get container status \"291eaad44d9b316ab0780c3a4a51f856d4a6a17ef5d64bef7fe33dc7319ff37b\": rpc error: code = NotFound desc = could not find container \"291eaad44d9b316ab0780c3a4a51f856d4a6a17ef5d64bef7fe33dc7319ff37b\": container with ID starting with 291eaad44d9b316ab0780c3a4a51f856d4a6a17ef5d64bef7fe33dc7319ff37b not found: ID does not exist"
Sep 16 17:54:15.119168 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:15.119142    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"]
Sep 16 17:54:15.121260 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:15.121237    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-7555c9bf99-njnhl"]
Sep 16 17:54:17.081134 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:17.081104    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="525daaaf-c2af-4bf8-bf27-d777c52a5147" path="/var/lib/kubelet/pods/525daaaf-c2af-4bf8-bf27-d777c52a5147/volumes"
Sep 16 17:54:48.914343 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:48.914310    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:54:49.025809 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:49.025770    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:54:49.157404 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:49.157367    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:54:55.819224 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:55.819190    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:54:56.966076 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:56.966038    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc"]
Sep 16 17:54:56.966453 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:56.966386    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="525daaaf-c2af-4bf8-bf27-d777c52a5147" containerName="storage-initializer"
Sep 16 17:54:56.966453 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:56.966406    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="525daaaf-c2af-4bf8-bf27-d777c52a5147" containerName="storage-initializer"
Sep 16 17:54:56.966453 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:56.966416    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="525daaaf-c2af-4bf8-bf27-d777c52a5147" containerName="main"
Sep 16 17:54:56.966453 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:56.966421    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="525daaaf-c2af-4bf8-bf27-d777c52a5147" containerName="main"
Sep 16 17:54:56.966453 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:56.966437    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fa789b8-bd57-46b8-84a1-ac378bd5a32e" containerName="main"
Sep 16 17:54:56.966453 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:56.966444    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fa789b8-bd57-46b8-84a1-ac378bd5a32e" containerName="main"
Sep 16 17:54:56.966658 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:56.966503    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="525daaaf-c2af-4bf8-bf27-d777c52a5147" containerName="main"
Sep 16 17:54:56.966658 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:56.966510    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fa789b8-bd57-46b8-84a1-ac378bd5a32e" containerName="main"
Sep 16 17:54:56.969561 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:56.969544    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc"
Sep 16 17:54:56.973942 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:56.973910    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"openshift-service-ca.crt\""
Sep 16 17:54:56.973942 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:56.973923    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v1-kserve-self-signed-certs\""
Sep 16 17:54:56.973942 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:56.973935    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"kube-root-ca.crt\""
Sep 16 17:54:56.974127 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:56.973924    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"default-dockercfg-dzmvw\""
Sep 16 17:54:56.985406 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:56.985383    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc"]
Sep 16 17:54:57.057858 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:57.057824    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pxjv\" (UniqueName: \"kubernetes.io/projected/da4aca3f-af68-4c0f-849a-6b22879ccef0-kube-api-access-6pxjv\") pod \"canary-v1-kserve-65d58ccc9-vvgbc\" (UID: \"da4aca3f-af68-4c0f-849a-6b22879ccef0\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc"
Sep 16 17:54:57.057858 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:57.057876    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da4aca3f-af68-4c0f-849a-6b22879ccef0-home\") pod \"canary-v1-kserve-65d58ccc9-vvgbc\" (UID: \"da4aca3f-af68-4c0f-849a-6b22879ccef0\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc"
Sep 16 17:54:57.058066 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:57.057942    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4aca3f-af68-4c0f-849a-6b22879ccef0-dshm\") pod \"canary-v1-kserve-65d58ccc9-vvgbc\" (UID: \"da4aca3f-af68-4c0f-849a-6b22879ccef0\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc"
Sep 16 17:54:57.058066 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:57.057991    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4aca3f-af68-4c0f-849a-6b22879ccef0-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-vvgbc\" (UID: \"da4aca3f-af68-4c0f-849a-6b22879ccef0\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc"
Sep 16 17:54:57.058066 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:57.058010    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4aca3f-af68-4c0f-849a-6b22879ccef0-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-vvgbc\" (UID: \"da4aca3f-af68-4c0f-849a-6b22879ccef0\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc"
Sep 16 17:54:57.058169 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:57.058072    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4aca3f-af68-4c0f-849a-6b22879ccef0-model-cache\") pod \"canary-v1-kserve-65d58ccc9-vvgbc\" (UID: \"da4aca3f-af68-4c0f-849a-6b22879ccef0\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc"
Sep 16 17:54:57.063509 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:57.063482    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:54:57.159301 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:57.159258    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4aca3f-af68-4c0f-849a-6b22879ccef0-dshm\") pod \"canary-v1-kserve-65d58ccc9-vvgbc\" (UID: \"da4aca3f-af68-4c0f-849a-6b22879ccef0\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc"
Sep 16 17:54:57.159494 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:57.159318    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4aca3f-af68-4c0f-849a-6b22879ccef0-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-vvgbc\" (UID: \"da4aca3f-af68-4c0f-849a-6b22879ccef0\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc"
Sep 16 17:54:57.159494 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:57.159345    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4aca3f-af68-4c0f-849a-6b22879ccef0-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-vvgbc\" (UID: \"da4aca3f-af68-4c0f-849a-6b22879ccef0\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc"
Sep 16 17:54:57.159494 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:57.159387    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4aca3f-af68-4c0f-849a-6b22879ccef0-model-cache\") pod \"canary-v1-kserve-65d58ccc9-vvgbc\" (UID: \"da4aca3f-af68-4c0f-849a-6b22879ccef0\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc"
Sep 16 17:54:57.159494 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:57.159471    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pxjv\" (UniqueName: \"kubernetes.io/projected/da4aca3f-af68-4c0f-849a-6b22879ccef0-kube-api-access-6pxjv\") pod \"canary-v1-kserve-65d58ccc9-vvgbc\" (UID: \"da4aca3f-af68-4c0f-849a-6b22879ccef0\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc"
Sep 16 17:54:57.159724 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:57.159500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da4aca3f-af68-4c0f-849a-6b22879ccef0-home\") pod \"canary-v1-kserve-65d58ccc9-vvgbc\" (UID: \"da4aca3f-af68-4c0f-849a-6b22879ccef0\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc"
Sep 16 17:54:57.159798 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:57.159764    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4aca3f-af68-4c0f-849a-6b22879ccef0-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-vvgbc\" (UID: \"da4aca3f-af68-4c0f-849a-6b22879ccef0\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc"
Sep 16 17:54:57.159851 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:57.159812    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4aca3f-af68-4c0f-849a-6b22879ccef0-model-cache\") pod \"canary-v1-kserve-65d58ccc9-vvgbc\" (UID: \"da4aca3f-af68-4c0f-849a-6b22879ccef0\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc"
Sep 16 17:54:57.159939 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:57.159901    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da4aca3f-af68-4c0f-849a-6b22879ccef0-home\") pod \"canary-v1-kserve-65d58ccc9-vvgbc\" (UID: \"da4aca3f-af68-4c0f-849a-6b22879ccef0\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc"
Sep 16 17:54:57.161645 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:57.161622    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4aca3f-af68-4c0f-849a-6b22879ccef0-dshm\") pod \"canary-v1-kserve-65d58ccc9-vvgbc\" (UID: \"da4aca3f-af68-4c0f-849a-6b22879ccef0\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc"
Sep 16 17:54:57.161989 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:57.161970    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4aca3f-af68-4c0f-849a-6b22879ccef0-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-vvgbc\" (UID: \"da4aca3f-af68-4c0f-849a-6b22879ccef0\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc"
Sep 16 17:54:57.169065 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:57.169041    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pxjv\" (UniqueName: \"kubernetes.io/projected/da4aca3f-af68-4c0f-849a-6b22879ccef0-kube-api-access-6pxjv\") pod \"canary-v1-kserve-65d58ccc9-vvgbc\" (UID: \"da4aca3f-af68-4c0f-849a-6b22879ccef0\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc"
Sep 16 17:54:57.279949 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:57.279831    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc"
Sep 16 17:54:57.576954 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:57.576926    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc"]
Sep 16 17:54:57.579328 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:54:57.579303    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4aca3f_af68_4c0f_849a_6b22879ccef0.slice/crio-3a22d9148fab991cf08ec94c4c6e2e8b7f1f08e2f6c1894270a63047897292a3 WatchSource:0}: Error finding container 3a22d9148fab991cf08ec94c4c6e2e8b7f1f08e2f6c1894270a63047897292a3: Status 404 returned error can't find the container with id 3a22d9148fab991cf08ec94c4c6e2e8b7f1f08e2f6c1894270a63047897292a3
Sep 16 17:54:57.581574 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:57.581555    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 17:54:58.304990 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:58.304950    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc" event={"ID":"da4aca3f-af68-4c0f-849a-6b22879ccef0","Type":"ContainerStarted","Data":"3a22d9148fab991cf08ec94c4c6e2e8b7f1f08e2f6c1894270a63047897292a3"}
Sep 16 17:54:59.311907 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:59.311852    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc" event={"ID":"da4aca3f-af68-4c0f-849a-6b22879ccef0","Type":"ContainerStarted","Data":"ddbf04207938e35a96c816308544a7289101e1e7be6c636da9250c6befdb89f3"}
Sep 16 17:54:59.335832 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:54:59.335782    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc" podStartSLOduration=2.383020298 podStartE2EDuration="3.335764624s" totalImagesPullingTime="952.744326ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:54:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:54:59.334805998 +0000 UTC m=+1820.697484827" watchObservedRunningTime="2026-09-16 17:54:59.335764624 +0000 UTC m=+1820.698443455"
Sep 16 17:55:00.250368 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:00.250330    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb"]
Sep 16 17:55:00.254008 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:00.253982    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb"
Sep 16 17:55:00.257348 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:00.257329    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v2-kserve-self-signed-certs\""
Sep 16 17:55:00.266766 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:00.266740    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb"]
Sep 16 17:55:00.280385 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:00.280354    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc"
Sep 16 17:55:00.295610 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:00.295577    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:55:00.389197 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:00.389155    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqpn5\" (UniqueName: \"kubernetes.io/projected/0a83b189-2080-48d5-adcc-66f9b72f547f-kube-api-access-sqpn5\") pod \"canary-v2-kserve-7f65dbbccc-slqsb\" (UID: \"0a83b189-2080-48d5-adcc-66f9b72f547f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb"
Sep 16 17:55:00.389590 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:00.389208    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a83b189-2080-48d5-adcc-66f9b72f547f-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-slqsb\" (UID: \"0a83b189-2080-48d5-adcc-66f9b72f547f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb"
Sep 16 17:55:00.389590 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:00.389252    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a83b189-2080-48d5-adcc-66f9b72f547f-home\") pod \"canary-v2-kserve-7f65dbbccc-slqsb\" (UID: \"0a83b189-2080-48d5-adcc-66f9b72f547f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb"
Sep 16 17:55:00.389590 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:00.389322    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a83b189-2080-48d5-adcc-66f9b72f547f-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-slqsb\" (UID: \"0a83b189-2080-48d5-adcc-66f9b72f547f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb"
Sep 16 17:55:00.389590 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:00.389349    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a83b189-2080-48d5-adcc-66f9b72f547f-dshm\") pod \"canary-v2-kserve-7f65dbbccc-slqsb\" (UID: \"0a83b189-2080-48d5-adcc-66f9b72f547f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb"
Sep 16 17:55:00.389590 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:00.389377    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a83b189-2080-48d5-adcc-66f9b72f547f-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-slqsb\" (UID: \"0a83b189-2080-48d5-adcc-66f9b72f547f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb"
Sep 16 17:55:00.490810 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:00.490772    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a83b189-2080-48d5-adcc-66f9b72f547f-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-slqsb\" (UID: \"0a83b189-2080-48d5-adcc-66f9b72f547f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb"
Sep 16 17:55:00.491023 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:00.490824    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a83b189-2080-48d5-adcc-66f9b72f547f-dshm\") pod \"canary-v2-kserve-7f65dbbccc-slqsb\" (UID: \"0a83b189-2080-48d5-adcc-66f9b72f547f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb"
Sep 16 17:55:00.491023 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:00.490874    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a83b189-2080-48d5-adcc-66f9b72f547f-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-slqsb\" (UID: \"0a83b189-2080-48d5-adcc-66f9b72f547f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb"
Sep 16 17:55:00.491023 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:00.490952    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqpn5\" (UniqueName: \"kubernetes.io/projected/0a83b189-2080-48d5-adcc-66f9b72f547f-kube-api-access-sqpn5\") pod \"canary-v2-kserve-7f65dbbccc-slqsb\" (UID: \"0a83b189-2080-48d5-adcc-66f9b72f547f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb"
Sep 16 17:55:00.491023 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:00.490997    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a83b189-2080-48d5-adcc-66f9b72f547f-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-slqsb\" (UID: \"0a83b189-2080-48d5-adcc-66f9b72f547f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb"
Sep 16 17:55:00.491253 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:00.491023    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a83b189-2080-48d5-adcc-66f9b72f547f-home\") pod \"canary-v2-kserve-7f65dbbccc-slqsb\" (UID: \"0a83b189-2080-48d5-adcc-66f9b72f547f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb"
Sep 16 17:55:00.491316 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:00.491253    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a83b189-2080-48d5-adcc-66f9b72f547f-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-slqsb\" (UID: \"0a83b189-2080-48d5-adcc-66f9b72f547f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb"
Sep 16 17:55:00.491316 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:00.491279    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a83b189-2080-48d5-adcc-66f9b72f547f-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-slqsb\" (UID: \"0a83b189-2080-48d5-adcc-66f9b72f547f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb"
Sep 16 17:55:00.491416 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:00.491336    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a83b189-2080-48d5-adcc-66f9b72f547f-home\") pod \"canary-v2-kserve-7f65dbbccc-slqsb\" (UID: \"0a83b189-2080-48d5-adcc-66f9b72f547f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb"
Sep 16 17:55:00.493576 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:00.493546    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a83b189-2080-48d5-adcc-66f9b72f547f-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-slqsb\" (UID: \"0a83b189-2080-48d5-adcc-66f9b72f547f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb"
Sep 16 17:55:00.493721 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:00.493593    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a83b189-2080-48d5-adcc-66f9b72f547f-dshm\") pod \"canary-v2-kserve-7f65dbbccc-slqsb\" (UID: \"0a83b189-2080-48d5-adcc-66f9b72f547f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb"
Sep 16 17:55:00.499613 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:00.499589    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqpn5\" (UniqueName: \"kubernetes.io/projected/0a83b189-2080-48d5-adcc-66f9b72f547f-kube-api-access-sqpn5\") pod \"canary-v2-kserve-7f65dbbccc-slqsb\" (UID: \"0a83b189-2080-48d5-adcc-66f9b72f547f\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb"
Sep 16 17:55:00.564431 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:00.564407    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb"
Sep 16 17:55:00.869164 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:00.869133    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb"]
Sep 16 17:55:00.869389 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:55:00.869369    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a83b189_2080_48d5_adcc_66f9b72f547f.slice/crio-0a098bb396b3ce752731872f8eddb5c74bb42645148b2df2d7a0074ada164de9 WatchSource:0}: Error finding container 0a098bb396b3ce752731872f8eddb5c74bb42645148b2df2d7a0074ada164de9: Status 404 returned error can't find the container with id 0a098bb396b3ce752731872f8eddb5c74bb42645148b2df2d7a0074ada164de9
Sep 16 17:55:00.894942 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:00.894911    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:55:01.027745 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:01.027708    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:55:01.323648 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:01.323610    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb" event={"ID":"0a83b189-2080-48d5-adcc-66f9b72f547f","Type":"ContainerStarted","Data":"04346c9775c253cecd05a99f0b69db65444afbfbc3a8c40629b0de39cd35af0a"}
Sep 16 17:55:01.323847 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:01.323677    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb" event={"ID":"0a83b189-2080-48d5-adcc-66f9b72f547f","Type":"ContainerStarted","Data":"0a098bb396b3ce752731872f8eddb5c74bb42645148b2df2d7a0074ada164de9"}
Sep 16 17:55:01.398205 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:01.398152    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb" podStartSLOduration=1.39813586 podStartE2EDuration="1.39813586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:55:01.394832338 +0000 UTC m=+1822.757511123" watchObservedRunningTime="2026-09-16 17:55:01.39813586 +0000 UTC m=+1822.760814646"
Sep 16 17:55:01.565097 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:01.565065    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb"
Sep 16 17:55:03.576901 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:03.576855    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:55:03.683759 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:03.683719    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:55:07.280370 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:07.280330    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc"
Sep 16 17:55:07.295900 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:07.295850    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc"
Sep 16 17:55:07.365548 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:07.365519    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc"
Sep 16 17:55:07.668096 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:07.668063    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:55:10.565077 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:10.565034    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb"
Sep 16 17:55:10.577526 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:10.577501    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb"
Sep 16 17:55:11.386429 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:11.386398    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb"
Sep 16 17:55:13.847489 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:13.847450    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:55:13.935686 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:13.935651    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:55:20.495397 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:20.495365    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"]
Sep 16 17:55:20.498744 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:20.498722    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"
Sep 16 17:55:20.502624 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:20.502602    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-vtflj\""
Sep 16 17:55:20.503439 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:20.503411    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 16 17:55:20.503722 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:20.503699    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-qxhkt\""
Sep 16 17:55:20.503844 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:20.503813    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 16 17:55:20.503948 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:20.503858    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 16 17:55:20.519437 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:20.519409    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"]
Sep 16 17:55:20.556309 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:20.556272    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b\" (UID: \"6cdfa8bb-389b-4420-9883-1793ffcfb3a2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"
Sep 16 17:55:20.556467 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:20.556316    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b\" (UID: \"6cdfa8bb-389b-4420-9883-1793ffcfb3a2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"
Sep 16 17:55:20.556467 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:20.556371    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b\" (UID: \"6cdfa8bb-389b-4420-9883-1793ffcfb3a2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"
Sep 16 17:55:20.556467 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:20.556411    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzjv7\" (UniqueName: \"kubernetes.io/projected/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-kube-api-access-zzjv7\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b\" (UID: \"6cdfa8bb-389b-4420-9883-1793ffcfb3a2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"
Sep 16 17:55:20.556467 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:20.556451    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b\" (UID: \"6cdfa8bb-389b-4420-9883-1793ffcfb3a2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"
Sep 16 17:55:20.556607 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:20.556492    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b\" (UID: \"6cdfa8bb-389b-4420-9883-1793ffcfb3a2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"
Sep 16 17:55:20.563412 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:20.563377    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:55:20.657373 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:20.657332    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b\" (UID: \"6cdfa8bb-389b-4420-9883-1793ffcfb3a2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"
Sep 16 17:55:20.657373 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:20.657373    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b\" (UID: \"6cdfa8bb-389b-4420-9883-1793ffcfb3a2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"
Sep 16 17:55:20.657594 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:20.657397    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzjv7\" (UniqueName: \"kubernetes.io/projected/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-kube-api-access-zzjv7\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b\" (UID: \"6cdfa8bb-389b-4420-9883-1793ffcfb3a2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"
Sep 16 17:55:20.657594 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:20.657432    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b\" (UID: \"6cdfa8bb-389b-4420-9883-1793ffcfb3a2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"
Sep 16 17:55:20.657594 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:20.657456    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b\" (UID: \"6cdfa8bb-389b-4420-9883-1793ffcfb3a2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"
Sep 16 17:55:20.657594 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:20.657517    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b\" (UID: \"6cdfa8bb-389b-4420-9883-1793ffcfb3a2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"
Sep 16 17:55:20.657928 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:20.657905    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b\" (UID: \"6cdfa8bb-389b-4420-9883-1793ffcfb3a2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"
Sep 16 17:55:20.658032 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:20.657932    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b\" (UID: \"6cdfa8bb-389b-4420-9883-1793ffcfb3a2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"
Sep 16 17:55:20.658032 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:20.657959    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b\" (UID: \"6cdfa8bb-389b-4420-9883-1793ffcfb3a2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"
Sep 16 17:55:20.659750 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:20.659721    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b\" (UID: \"6cdfa8bb-389b-4420-9883-1793ffcfb3a2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"
Sep 16 17:55:20.660026 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:20.660009    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b\" (UID: \"6cdfa8bb-389b-4420-9883-1793ffcfb3a2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"
Sep 16 17:55:20.666683 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:20.666652    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzjv7\" (UniqueName: \"kubernetes.io/projected/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-kube-api-access-zzjv7\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b\" (UID: \"6cdfa8bb-389b-4420-9883-1793ffcfb3a2\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"
Sep 16 17:55:20.809184 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:20.809105    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"
Sep 16 17:55:20.819090 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:20.819047    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:55:21.093301 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:21.093276    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"]
Sep 16 17:55:21.095598 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:55:21.095568    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cdfa8bb_389b_4420_9883_1793ffcfb3a2.slice/crio-0c0dc4e1b4b8399b41e463385c1e7da25b0488f28fad96b84b22665aabeee63c WatchSource:0}: Error finding container 0c0dc4e1b4b8399b41e463385c1e7da25b0488f28fad96b84b22665aabeee63c: Status 404 returned error can't find the container with id 0c0dc4e1b4b8399b41e463385c1e7da25b0488f28fad96b84b22665aabeee63c
Sep 16 17:55:21.434692 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:21.434601    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b" event={"ID":"6cdfa8bb-389b-4420-9883-1793ffcfb3a2","Type":"ContainerStarted","Data":"0c0dc4e1b4b8399b41e463385c1e7da25b0488f28fad96b84b22665aabeee63c"}
Sep 16 17:55:23.448080 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:23.448033    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b" event={"ID":"6cdfa8bb-389b-4420-9883-1793ffcfb3a2","Type":"ContainerStarted","Data":"dc06b99f3c098b905edc6c1f7c4ad1be2a182efefb415ab2dc72d95c2ef31fa0"}
Sep 16 17:55:23.448470 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:23.448173    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"
Sep 16 17:55:24.455077 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:24.455039    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b" event={"ID":"6cdfa8bb-389b-4420-9883-1793ffcfb3a2","Type":"ContainerStarted","Data":"ba238eaf4c5a8d1f867944b7e530735ad66c8f78d8942c56f085c72199087105"}
Sep 16 17:55:24.810211 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:24.810182    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"
Sep 16 17:55:30.809433 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:30.809379    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"
Sep 16 17:55:30.821359 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:30.821331    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"
Sep 16 17:55:30.848365 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:30.848230    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b" podStartSLOduration=8.68178238 podStartE2EDuration="10.848211068s" totalImagesPullingTime="2.166428688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:55:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:55:24.488687827 +0000 UTC m=+1845.851366613" watchObservedRunningTime="2026-09-16 17:55:30.848211068 +0000 UTC m=+1852.210889855"
Sep 16 17:55:31.502419 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:31.502382    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"
Sep 16 17:55:36.948478 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:36.948435    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:55:42.508382 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:55:42.508352    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"
Sep 16 17:56:24.935997 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:56:24.935957    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:56:30.305819 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:56:30.305774    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:56:30.421964 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:56:30.421933    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:56:43.254851 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:56:43.254808    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:57:00.968689 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:00.968635    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:57:01.242236 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:01.242154    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc"]
Sep 16 17:57:01.288100 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:01.288065    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:57:01.553976 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:01.553883    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:57:01.701190 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:01.701141    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb"]
Sep 16 17:57:01.920838 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:01.920797    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:57:05.899881 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:05.899824    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:57:07.999168 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:07.999131    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:57:16.261166 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:16.261126    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc" podUID="da4aca3f-af68-4c0f-849a-6b22879ccef0" containerName="main" containerID="cri-o://ddbf04207938e35a96c816308544a7289101e1e7be6c636da9250c6befdb89f3" gracePeriod=45
Sep 16 17:57:16.354618 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:16.354580    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc" podUID="da4aca3f-af68-4c0f-849a-6b22879ccef0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Sep 16 17:57:16.513155 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:16.513093    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc"
Sep 16 17:57:16.554075 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:16.554044    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4aca3f-af68-4c0f-849a-6b22879ccef0-tmp-dir\") pod \"da4aca3f-af68-4c0f-849a-6b22879ccef0\" (UID: \"da4aca3f-af68-4c0f-849a-6b22879ccef0\") "
Sep 16 17:57:16.554238 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:16.554092    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da4aca3f-af68-4c0f-849a-6b22879ccef0-home\") pod \"da4aca3f-af68-4c0f-849a-6b22879ccef0\" (UID: \"da4aca3f-af68-4c0f-849a-6b22879ccef0\") "
Sep 16 17:57:16.554238 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:16.554127    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4aca3f-af68-4c0f-849a-6b22879ccef0-tls-certs\") pod \"da4aca3f-af68-4c0f-849a-6b22879ccef0\" (UID: \"da4aca3f-af68-4c0f-849a-6b22879ccef0\") "
Sep 16 17:57:16.554238 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:16.554153    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4aca3f-af68-4c0f-849a-6b22879ccef0-model-cache\") pod \"da4aca3f-af68-4c0f-849a-6b22879ccef0\" (UID: \"da4aca3f-af68-4c0f-849a-6b22879ccef0\") "
Sep 16 17:57:16.554238 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:16.554179    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4aca3f-af68-4c0f-849a-6b22879ccef0-dshm\") pod \"da4aca3f-af68-4c0f-849a-6b22879ccef0\" (UID: \"da4aca3f-af68-4c0f-849a-6b22879ccef0\") "
Sep 16 17:57:16.554238 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:16.554239    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pxjv\" (UniqueName: \"kubernetes.io/projected/da4aca3f-af68-4c0f-849a-6b22879ccef0-kube-api-access-6pxjv\") pod \"da4aca3f-af68-4c0f-849a-6b22879ccef0\" (UID: \"da4aca3f-af68-4c0f-849a-6b22879ccef0\") "
Sep 16 17:57:16.554512 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:16.554363    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4aca3f-af68-4c0f-849a-6b22879ccef0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "da4aca3f-af68-4c0f-849a-6b22879ccef0" (UID: "da4aca3f-af68-4c0f-849a-6b22879ccef0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:57:16.554512 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:16.554458    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4aca3f-af68-4c0f-849a-6b22879ccef0-home" (OuterVolumeSpecName: "home") pod "da4aca3f-af68-4c0f-849a-6b22879ccef0" (UID: "da4aca3f-af68-4c0f-849a-6b22879ccef0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:57:16.554512 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:16.554469    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da4aca3f-af68-4c0f-849a-6b22879ccef0-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:57:16.554512 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:16.554476    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4aca3f-af68-4c0f-849a-6b22879ccef0-model-cache" (OuterVolumeSpecName: "model-cache") pod "da4aca3f-af68-4c0f-849a-6b22879ccef0" (UID: "da4aca3f-af68-4c0f-849a-6b22879ccef0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:57:16.556376 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:16.556339    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da4aca3f-af68-4c0f-849a-6b22879ccef0-dshm" (OuterVolumeSpecName: "dshm") pod "da4aca3f-af68-4c0f-849a-6b22879ccef0" (UID: "da4aca3f-af68-4c0f-849a-6b22879ccef0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:57:16.556479 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:16.556375    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da4aca3f-af68-4c0f-849a-6b22879ccef0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "da4aca3f-af68-4c0f-849a-6b22879ccef0" (UID: "da4aca3f-af68-4c0f-849a-6b22879ccef0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:57:16.556479 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:16.556455    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da4aca3f-af68-4c0f-849a-6b22879ccef0-kube-api-access-6pxjv" (OuterVolumeSpecName: "kube-api-access-6pxjv") pod "da4aca3f-af68-4c0f-849a-6b22879ccef0" (UID: "da4aca3f-af68-4c0f-849a-6b22879ccef0"). InnerVolumeSpecName "kube-api-access-6pxjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:57:16.654948 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:16.654912    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da4aca3f-af68-4c0f-849a-6b22879ccef0-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:57:16.654948 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:16.654943    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da4aca3f-af68-4c0f-849a-6b22879ccef0-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:57:16.654948 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:16.654952    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pxjv\" (UniqueName: \"kubernetes.io/projected/da4aca3f-af68-4c0f-849a-6b22879ccef0-kube-api-access-6pxjv\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:57:16.655191 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:16.654961    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da4aca3f-af68-4c0f-849a-6b22879ccef0-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:57:16.655191 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:16.654974    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da4aca3f-af68-4c0f-849a-6b22879ccef0-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:57:16.720133 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:16.720070    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb" podUID="0a83b189-2080-48d5-adcc-66f9b72f547f" containerName="main" containerID="cri-o://04346c9775c253cecd05a99f0b69db65444afbfbc3a8c40629b0de39cd35af0a" gracePeriod=45
Sep 16 17:57:16.966139 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:16.966115    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb"
Sep 16 17:57:17.058662 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.058631    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a83b189-2080-48d5-adcc-66f9b72f547f-tls-certs\") pod \"0a83b189-2080-48d5-adcc-66f9b72f547f\" (UID: \"0a83b189-2080-48d5-adcc-66f9b72f547f\") "
Sep 16 17:57:17.058859 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.058673    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a83b189-2080-48d5-adcc-66f9b72f547f-dshm\") pod \"0a83b189-2080-48d5-adcc-66f9b72f547f\" (UID: \"0a83b189-2080-48d5-adcc-66f9b72f547f\") "
Sep 16 17:57:17.058859 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.058691    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a83b189-2080-48d5-adcc-66f9b72f547f-tmp-dir\") pod \"0a83b189-2080-48d5-adcc-66f9b72f547f\" (UID: \"0a83b189-2080-48d5-adcc-66f9b72f547f\") "
Sep 16 17:57:17.058859 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.058742    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a83b189-2080-48d5-adcc-66f9b72f547f-model-cache\") pod \"0a83b189-2080-48d5-adcc-66f9b72f547f\" (UID: \"0a83b189-2080-48d5-adcc-66f9b72f547f\") "
Sep 16 17:57:17.058859 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.058782    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqpn5\" (UniqueName: \"kubernetes.io/projected/0a83b189-2080-48d5-adcc-66f9b72f547f-kube-api-access-sqpn5\") pod \"0a83b189-2080-48d5-adcc-66f9b72f547f\" (UID: \"0a83b189-2080-48d5-adcc-66f9b72f547f\") "
Sep 16 17:57:17.058859 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.058830    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a83b189-2080-48d5-adcc-66f9b72f547f-home\") pod \"0a83b189-2080-48d5-adcc-66f9b72f547f\" (UID: \"0a83b189-2080-48d5-adcc-66f9b72f547f\") "
Sep 16 17:57:17.059147 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.059077    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a83b189-2080-48d5-adcc-66f9b72f547f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0a83b189-2080-48d5-adcc-66f9b72f547f" (UID: "0a83b189-2080-48d5-adcc-66f9b72f547f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:57:17.059147 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.059086    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a83b189-2080-48d5-adcc-66f9b72f547f-model-cache" (OuterVolumeSpecName: "model-cache") pod "0a83b189-2080-48d5-adcc-66f9b72f547f" (UID: "0a83b189-2080-48d5-adcc-66f9b72f547f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:57:17.059340 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.059319    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a83b189-2080-48d5-adcc-66f9b72f547f-home" (OuterVolumeSpecName: "home") pod "0a83b189-2080-48d5-adcc-66f9b72f547f" (UID: "0a83b189-2080-48d5-adcc-66f9b72f547f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:57:17.060900 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.060878    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a83b189-2080-48d5-adcc-66f9b72f547f-dshm" (OuterVolumeSpecName: "dshm") pod "0a83b189-2080-48d5-adcc-66f9b72f547f" (UID: "0a83b189-2080-48d5-adcc-66f9b72f547f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:57:17.060970 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.060939    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a83b189-2080-48d5-adcc-66f9b72f547f-kube-api-access-sqpn5" (OuterVolumeSpecName: "kube-api-access-sqpn5") pod "0a83b189-2080-48d5-adcc-66f9b72f547f" (UID: "0a83b189-2080-48d5-adcc-66f9b72f547f"). InnerVolumeSpecName "kube-api-access-sqpn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:57:17.060970 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.060939    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a83b189-2080-48d5-adcc-66f9b72f547f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0a83b189-2080-48d5-adcc-66f9b72f547f" (UID: "0a83b189-2080-48d5-adcc-66f9b72f547f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:57:17.074526 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.074493    2577 generic.go:356] "Generic (PLEG): container finished" podID="da4aca3f-af68-4c0f-849a-6b22879ccef0" containerID="ddbf04207938e35a96c816308544a7289101e1e7be6c636da9250c6befdb89f3" exitCode=0
Sep 16 17:57:17.074643 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.074607    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc"
Sep 16 17:57:17.076053 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.076033    2577 generic.go:356] "Generic (PLEG): container finished" podID="0a83b189-2080-48d5-adcc-66f9b72f547f" containerID="04346c9775c253cecd05a99f0b69db65444afbfbc3a8c40629b0de39cd35af0a" exitCode=0
Sep 16 17:57:17.076168 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.076114    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb"
Sep 16 17:57:17.090689 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.090037    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc" event={"ID":"da4aca3f-af68-4c0f-849a-6b22879ccef0","Type":"ContainerDied","Data":"ddbf04207938e35a96c816308544a7289101e1e7be6c636da9250c6befdb89f3"}
Sep 16 17:57:17.090689 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.090088    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc" event={"ID":"da4aca3f-af68-4c0f-849a-6b22879ccef0","Type":"ContainerDied","Data":"3a22d9148fab991cf08ec94c4c6e2e8b7f1f08e2f6c1894270a63047897292a3"}
Sep 16 17:57:17.090689 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.090133    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb" event={"ID":"0a83b189-2080-48d5-adcc-66f9b72f547f","Type":"ContainerDied","Data":"04346c9775c253cecd05a99f0b69db65444afbfbc3a8c40629b0de39cd35af0a"}
Sep 16 17:57:17.090689 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.090165    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb" event={"ID":"0a83b189-2080-48d5-adcc-66f9b72f547f","Type":"ContainerDied","Data":"0a098bb396b3ce752731872f8eddb5c74bb42645148b2df2d7a0074ada164de9"}
Sep 16 17:57:17.090689 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.090201    2577 scope.go:117] "RemoveContainer" containerID="ddbf04207938e35a96c816308544a7289101e1e7be6c636da9250c6befdb89f3"
Sep 16 17:57:17.103474 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.103454    2577 scope.go:117] "RemoveContainer" containerID="ddbf04207938e35a96c816308544a7289101e1e7be6c636da9250c6befdb89f3"
Sep 16 17:57:17.103739 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:57:17.103715    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddbf04207938e35a96c816308544a7289101e1e7be6c636da9250c6befdb89f3\": container with ID starting with ddbf04207938e35a96c816308544a7289101e1e7be6c636da9250c6befdb89f3 not found: ID does not exist" containerID="ddbf04207938e35a96c816308544a7289101e1e7be6c636da9250c6befdb89f3"
Sep 16 17:57:17.103805 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.103745    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddbf04207938e35a96c816308544a7289101e1e7be6c636da9250c6befdb89f3"} err="failed to get container status \"ddbf04207938e35a96c816308544a7289101e1e7be6c636da9250c6befdb89f3\": rpc error: code = NotFound desc = could not find container \"ddbf04207938e35a96c816308544a7289101e1e7be6c636da9250c6befdb89f3\": container with ID starting with ddbf04207938e35a96c816308544a7289101e1e7be6c636da9250c6befdb89f3 not found: ID does not exist"
Sep 16 17:57:17.103805 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.103761    2577 scope.go:117] "RemoveContainer" containerID="04346c9775c253cecd05a99f0b69db65444afbfbc3a8c40629b0de39cd35af0a"
Sep 16 17:57:17.111746 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.111732    2577 scope.go:117] "RemoveContainer" containerID="04346c9775c253cecd05a99f0b69db65444afbfbc3a8c40629b0de39cd35af0a"
Sep 16 17:57:17.112099 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:57:17.112078    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04346c9775c253cecd05a99f0b69db65444afbfbc3a8c40629b0de39cd35af0a\": container with ID starting with 04346c9775c253cecd05a99f0b69db65444afbfbc3a8c40629b0de39cd35af0a not found: ID does not exist" containerID="04346c9775c253cecd05a99f0b69db65444afbfbc3a8c40629b0de39cd35af0a"
Sep 16 17:57:17.112176 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.112104    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04346c9775c253cecd05a99f0b69db65444afbfbc3a8c40629b0de39cd35af0a"} err="failed to get container status \"04346c9775c253cecd05a99f0b69db65444afbfbc3a8c40629b0de39cd35af0a\": rpc error: code = NotFound desc = could not find container \"04346c9775c253cecd05a99f0b69db65444afbfbc3a8c40629b0de39cd35af0a\": container with ID starting with 04346c9775c253cecd05a99f0b69db65444afbfbc3a8c40629b0de39cd35af0a not found: ID does not exist"
Sep 16 17:57:17.131847 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.131815    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc"]
Sep 16 17:57:17.133790 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.133766    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-vvgbc"]
Sep 16 17:57:17.154916 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.154886    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb"]
Sep 16 17:57:17.160121 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.160096    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a83b189-2080-48d5-adcc-66f9b72f547f-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:57:17.160121 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.160122    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a83b189-2080-48d5-adcc-66f9b72f547f-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:57:17.160297 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.160135    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a83b189-2080-48d5-adcc-66f9b72f547f-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:57:17.160297 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.160145    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqpn5\" (UniqueName: \"kubernetes.io/projected/0a83b189-2080-48d5-adcc-66f9b72f547f-kube-api-access-sqpn5\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:57:17.160297 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.160155    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a83b189-2080-48d5-adcc-66f9b72f547f-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:57:17.160297 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.160164    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a83b189-2080-48d5-adcc-66f9b72f547f-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:57:17.166647 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:17.166625    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-slqsb"]
Sep 16 17:57:19.082036 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:19.082005    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a83b189-2080-48d5-adcc-66f9b72f547f" path="/var/lib/kubelet/pods/0a83b189-2080-48d5-adcc-66f9b72f547f/volumes"
Sep 16 17:57:19.082402 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:19.082385    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da4aca3f-af68-4c0f-849a-6b22879ccef0" path="/var/lib/kubelet/pods/da4aca3f-af68-4c0f-849a-6b22879ccef0/volumes"
Sep 16 17:57:23.222622 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.222584    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf"]
Sep 16 17:57:23.223142 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.223120    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a83b189-2080-48d5-adcc-66f9b72f547f" containerName="main"
Sep 16 17:57:23.223142 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.223141    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a83b189-2080-48d5-adcc-66f9b72f547f" containerName="main"
Sep 16 17:57:23.223266 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.223155    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da4aca3f-af68-4c0f-849a-6b22879ccef0" containerName="main"
Sep 16 17:57:23.223266 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.223164    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="da4aca3f-af68-4c0f-849a-6b22879ccef0" containerName="main"
Sep 16 17:57:23.223365 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.223269    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a83b189-2080-48d5-adcc-66f9b72f547f" containerName="main"
Sep 16 17:57:23.223365 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.223284    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="da4aca3f-af68-4c0f-849a-6b22879ccef0" containerName="main"
Sep 16 17:57:23.227813 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.227791    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf"
Sep 16 17:57:23.231046 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.231029    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"default-dockercfg-n9fmz\""
Sep 16 17:57:23.232138 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.232120    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"kube-root-ca.crt\""
Sep 16 17:57:23.232296 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.232281    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"openshift-service-ca.crt\""
Sep 16 17:57:23.232413 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.232397    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v2-kserve-self-signed-certs\""
Sep 16 17:57:23.259206 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.259180    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf"]
Sep 16 17:57:23.309506 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.309472    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3ef9e25-1031-4fae-99db-ecdc76a10483-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-b2cxf\" (UID: \"d3ef9e25-1031-4fae-99db-ecdc76a10483\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf"
Sep 16 17:57:23.309697 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.309516    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3ef9e25-1031-4fae-99db-ecdc76a10483-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-b2cxf\" (UID: \"d3ef9e25-1031-4fae-99db-ecdc76a10483\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf"
Sep 16 17:57:23.309697 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.309600    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf4fm\" (UniqueName: \"kubernetes.io/projected/d3ef9e25-1031-4fae-99db-ecdc76a10483-kube-api-access-hf4fm\") pod \"test-div-v2-kserve-5f6fdb6b77-b2cxf\" (UID: \"d3ef9e25-1031-4fae-99db-ecdc76a10483\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf"
Sep 16 17:57:23.309697 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.309639    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ef9e25-1031-4fae-99db-ecdc76a10483-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-b2cxf\" (UID: \"d3ef9e25-1031-4fae-99db-ecdc76a10483\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf"
Sep 16 17:57:23.309697 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.309686    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3ef9e25-1031-4fae-99db-ecdc76a10483-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-b2cxf\" (UID: \"d3ef9e25-1031-4fae-99db-ecdc76a10483\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf"
Sep 16 17:57:23.309918 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.309739    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3ef9e25-1031-4fae-99db-ecdc76a10483-home\") pod \"test-div-v2-kserve-5f6fdb6b77-b2cxf\" (UID: \"d3ef9e25-1031-4fae-99db-ecdc76a10483\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf"
Sep 16 17:57:23.364181 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.364153    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:57:23.410801 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.410764    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3ef9e25-1031-4fae-99db-ecdc76a10483-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-b2cxf\" (UID: \"d3ef9e25-1031-4fae-99db-ecdc76a10483\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf"
Sep 16 17:57:23.411010 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.410806    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3ef9e25-1031-4fae-99db-ecdc76a10483-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-b2cxf\" (UID: \"d3ef9e25-1031-4fae-99db-ecdc76a10483\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf"
Sep 16 17:57:23.411010 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.410886    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf4fm\" (UniqueName: \"kubernetes.io/projected/d3ef9e25-1031-4fae-99db-ecdc76a10483-kube-api-access-hf4fm\") pod \"test-div-v2-kserve-5f6fdb6b77-b2cxf\" (UID: \"d3ef9e25-1031-4fae-99db-ecdc76a10483\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf"
Sep 16 17:57:23.411010 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.410917    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ef9e25-1031-4fae-99db-ecdc76a10483-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-b2cxf\" (UID: \"d3ef9e25-1031-4fae-99db-ecdc76a10483\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf"
Sep 16 17:57:23.411010 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.410961    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3ef9e25-1031-4fae-99db-ecdc76a10483-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-b2cxf\" (UID: \"d3ef9e25-1031-4fae-99db-ecdc76a10483\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf"
Sep 16 17:57:23.411204 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.411106    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3ef9e25-1031-4fae-99db-ecdc76a10483-home\") pod \"test-div-v2-kserve-5f6fdb6b77-b2cxf\" (UID: \"d3ef9e25-1031-4fae-99db-ecdc76a10483\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf"
Sep 16 17:57:23.411377 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.411351    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3ef9e25-1031-4fae-99db-ecdc76a10483-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-b2cxf\" (UID: \"d3ef9e25-1031-4fae-99db-ecdc76a10483\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf"
Sep 16 17:57:23.411497 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.411397    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3ef9e25-1031-4fae-99db-ecdc76a10483-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-b2cxf\" (UID: \"d3ef9e25-1031-4fae-99db-ecdc76a10483\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf"
Sep 16 17:57:23.414890 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.411490    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3ef9e25-1031-4fae-99db-ecdc76a10483-home\") pod \"test-div-v2-kserve-5f6fdb6b77-b2cxf\" (UID: \"d3ef9e25-1031-4fae-99db-ecdc76a10483\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf"
Sep 16 17:57:23.419127 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.419103    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3ef9e25-1031-4fae-99db-ecdc76a10483-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-b2cxf\" (UID: \"d3ef9e25-1031-4fae-99db-ecdc76a10483\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf"
Sep 16 17:57:23.419533 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.419514    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ef9e25-1031-4fae-99db-ecdc76a10483-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-b2cxf\" (UID: \"d3ef9e25-1031-4fae-99db-ecdc76a10483\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf"
Sep 16 17:57:23.423653 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.423635    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf4fm\" (UniqueName: \"kubernetes.io/projected/d3ef9e25-1031-4fae-99db-ecdc76a10483-kube-api-access-hf4fm\") pod \"test-div-v2-kserve-5f6fdb6b77-b2cxf\" (UID: \"d3ef9e25-1031-4fae-99db-ecdc76a10483\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf"
Sep 16 17:57:23.538118 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.538029    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf"
Sep 16 17:57:23.845621 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:23.845587    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf"]
Sep 16 17:57:23.849028 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:57:23.848996    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ef9e25_1031_4fae_99db_ecdc76a10483.slice/crio-12a7ab49ecaf64442a08a870108fe2f59feee3aac580dd00125f4804f62cb3fb WatchSource:0}: Error finding container 12a7ab49ecaf64442a08a870108fe2f59feee3aac580dd00125f4804f62cb3fb: Status 404 returned error can't find the container with id 12a7ab49ecaf64442a08a870108fe2f59feee3aac580dd00125f4804f62cb3fb
Sep 16 17:57:24.117046 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:24.116946    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf" event={"ID":"d3ef9e25-1031-4fae-99db-ecdc76a10483","Type":"ContainerStarted","Data":"6997720c2ec5d630b582d7de1a565da65155fe14785dde4d2caceae13ddfb387"}
Sep 16 17:57:24.117046 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:24.117000    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf" event={"ID":"d3ef9e25-1031-4fae-99db-ecdc76a10483","Type":"ContainerStarted","Data":"12a7ab49ecaf64442a08a870108fe2f59feee3aac580dd00125f4804f62cb3fb"}
Sep 16 17:57:24.164713 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:24.164658    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf" podStartSLOduration=1.164640817 podStartE2EDuration="1.164640817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:57:24.15977121 +0000 UTC m=+1965.522450019" watchObservedRunningTime="2026-09-16 17:57:24.164640817 +0000 UTC m=+1965.527319603"
Sep 16 17:57:24.539095 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:24.539013    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf"
Sep 16 17:57:24.937390 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:24.937357    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"]
Sep 16 17:57:24.946448 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:57:24.946407    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 17:57:24.946448 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:57:24.946448 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 17:57:24.946448 ip-10-0-136-122 kubenswrapper[2577]:  > containerID="ba238eaf4c5a8d1f867944b7e530735ad66c8f78d8942c56f085c72199087105" cmd=["/bin/sleep","15"]
Sep 16 17:57:24.946707 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:24.946461    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b" message="" err=<
Sep 16 17:57:24.946707 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:57:24.946707 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 17:57:24.946707 ip-10-0-136-122 kubenswrapper[2577]:  >
Sep 16 17:57:24.946707 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:57:24.946473    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 17:57:24.946707 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:57:24.946707 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 17:57:24.946707 ip-10-0-136-122 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b" podUID="6cdfa8bb-389b-4420-9883-1793ffcfb3a2" containerName="main" containerID="cri-o://ba238eaf4c5a8d1f867944b7e530735ad66c8f78d8942c56f085c72199087105"
Sep 16 17:57:24.946707 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:24.946519    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b" podUID="6cdfa8bb-389b-4420-9883-1793ffcfb3a2" containerName="main" containerID="cri-o://ba238eaf4c5a8d1f867944b7e530735ad66c8f78d8942c56f085c72199087105" gracePeriod=60
Sep 16 17:57:24.970733 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:24.970702    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:57:25.124551 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:25.124520    2577 generic.go:356] "Generic (PLEG): container finished" podID="6cdfa8bb-389b-4420-9883-1793ffcfb3a2" containerID="ba238eaf4c5a8d1f867944b7e530735ad66c8f78d8942c56f085c72199087105" exitCode=0
Sep 16 17:57:25.124732 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:25.124600    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b" event={"ID":"6cdfa8bb-389b-4420-9883-1793ffcfb3a2","Type":"ContainerDied","Data":"ba238eaf4c5a8d1f867944b7e530735ad66c8f78d8942c56f085c72199087105"}
Sep 16 17:57:25.154721 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:25.154691    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:57:25.492034 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:25.491992    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b" podUID="6cdfa8bb-389b-4420-9883-1793ffcfb3a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8001/health\": dial tcp 10.133.0.61:8001: connect: connection refused"
Sep 16 17:57:25.559271 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:25.559240    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b" podUID="6cdfa8bb-389b-4420-9883-1793ffcfb3a2" containerName="llm-d-routing-sidecar" containerID="cri-o://dc06b99f3c098b905edc6c1f7c4ad1be2a182efefb415ab2dc72d95c2ef31fa0" gracePeriod=60
Sep 16 17:57:25.809875 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:25.809841    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"
Sep 16 17:57:25.930064 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:25.930030    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-tls-certs\") pod \"6cdfa8bb-389b-4420-9883-1793ffcfb3a2\" (UID: \"6cdfa8bb-389b-4420-9883-1793ffcfb3a2\") "
Sep 16 17:57:25.930270 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:25.930077    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzjv7\" (UniqueName: \"kubernetes.io/projected/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-kube-api-access-zzjv7\") pod \"6cdfa8bb-389b-4420-9883-1793ffcfb3a2\" (UID: \"6cdfa8bb-389b-4420-9883-1793ffcfb3a2\") "
Sep 16 17:57:25.930270 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:25.930122    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-tmp-dir\") pod \"6cdfa8bb-389b-4420-9883-1793ffcfb3a2\" (UID: \"6cdfa8bb-389b-4420-9883-1793ffcfb3a2\") "
Sep 16 17:57:25.930270 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:25.930177    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-dshm\") pod \"6cdfa8bb-389b-4420-9883-1793ffcfb3a2\" (UID: \"6cdfa8bb-389b-4420-9883-1793ffcfb3a2\") "
Sep 16 17:57:25.930270 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:25.930207    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-model-cache\") pod \"6cdfa8bb-389b-4420-9883-1793ffcfb3a2\" (UID: \"6cdfa8bb-389b-4420-9883-1793ffcfb3a2\") "
Sep 16 17:57:25.930270 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:25.930259    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-home\") pod \"6cdfa8bb-389b-4420-9883-1793ffcfb3a2\" (UID: \"6cdfa8bb-389b-4420-9883-1793ffcfb3a2\") "
Sep 16 17:57:25.930537 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:25.930409    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6cdfa8bb-389b-4420-9883-1793ffcfb3a2" (UID: "6cdfa8bb-389b-4420-9883-1793ffcfb3a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:57:25.930594 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:25.930555    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:57:25.930681 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:25.930629    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-home" (OuterVolumeSpecName: "home") pod "6cdfa8bb-389b-4420-9883-1793ffcfb3a2" (UID: "6cdfa8bb-389b-4420-9883-1793ffcfb3a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:57:25.930681 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:25.930686    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-model-cache" (OuterVolumeSpecName: "model-cache") pod "6cdfa8bb-389b-4420-9883-1793ffcfb3a2" (UID: "6cdfa8bb-389b-4420-9883-1793ffcfb3a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:57:25.932456 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:25.932427    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-dshm" (OuterVolumeSpecName: "dshm") pod "6cdfa8bb-389b-4420-9883-1793ffcfb3a2" (UID: "6cdfa8bb-389b-4420-9883-1793ffcfb3a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:57:25.932537 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:25.932461    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6cdfa8bb-389b-4420-9883-1793ffcfb3a2" (UID: "6cdfa8bb-389b-4420-9883-1793ffcfb3a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:57:25.932537 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:25.932503    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-kube-api-access-zzjv7" (OuterVolumeSpecName: "kube-api-access-zzjv7") pod "6cdfa8bb-389b-4420-9883-1793ffcfb3a2" (UID: "6cdfa8bb-389b-4420-9883-1793ffcfb3a2"). InnerVolumeSpecName "kube-api-access-zzjv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:57:26.031913 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:26.031851    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:57:26.031913 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:26.031907    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:57:26.031913 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:26.031918    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzjv7\" (UniqueName: \"kubernetes.io/projected/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-kube-api-access-zzjv7\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:57:26.032148 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:26.031928    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:57:26.032148 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:26.031937    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cdfa8bb-389b-4420-9883-1793ffcfb3a2-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:57:26.132746 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:26.132711    2577 generic.go:356] "Generic (PLEG): container finished" podID="6cdfa8bb-389b-4420-9883-1793ffcfb3a2" containerID="dc06b99f3c098b905edc6c1f7c4ad1be2a182efefb415ab2dc72d95c2ef31fa0" exitCode=0
Sep 16 17:57:26.132945 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:26.132764    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"
Sep 16 17:57:26.132945 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:26.132761    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b" event={"ID":"6cdfa8bb-389b-4420-9883-1793ffcfb3a2","Type":"ContainerDied","Data":"dc06b99f3c098b905edc6c1f7c4ad1be2a182efefb415ab2dc72d95c2ef31fa0"}
Sep 16 17:57:26.132945 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:26.132832    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b" event={"ID":"6cdfa8bb-389b-4420-9883-1793ffcfb3a2","Type":"ContainerDied","Data":"0c0dc4e1b4b8399b41e463385c1e7da25b0488f28fad96b84b22665aabeee63c"}
Sep 16 17:57:26.132945 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:26.132859    2577 scope.go:117] "RemoveContainer" containerID="ba238eaf4c5a8d1f867944b7e530735ad66c8f78d8942c56f085c72199087105"
Sep 16 17:57:26.144392 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:26.144368    2577 scope.go:117] "RemoveContainer" containerID="dc06b99f3c098b905edc6c1f7c4ad1be2a182efefb415ab2dc72d95c2ef31fa0"
Sep 16 17:57:26.153225 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:26.153201    2577 scope.go:117] "RemoveContainer" containerID="ba238eaf4c5a8d1f867944b7e530735ad66c8f78d8942c56f085c72199087105"
Sep 16 17:57:26.153500 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:57:26.153483    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba238eaf4c5a8d1f867944b7e530735ad66c8f78d8942c56f085c72199087105\": container with ID starting with ba238eaf4c5a8d1f867944b7e530735ad66c8f78d8942c56f085c72199087105 not found: ID does not exist" containerID="ba238eaf4c5a8d1f867944b7e530735ad66c8f78d8942c56f085c72199087105"
Sep 16 17:57:26.153549 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:26.153507    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba238eaf4c5a8d1f867944b7e530735ad66c8f78d8942c56f085c72199087105"} err="failed to get container status \"ba238eaf4c5a8d1f867944b7e530735ad66c8f78d8942c56f085c72199087105\": rpc error: code = NotFound desc = could not find container \"ba238eaf4c5a8d1f867944b7e530735ad66c8f78d8942c56f085c72199087105\": container with ID starting with ba238eaf4c5a8d1f867944b7e530735ad66c8f78d8942c56f085c72199087105 not found: ID does not exist"
Sep 16 17:57:26.153549 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:26.153524    2577 scope.go:117] "RemoveContainer" containerID="dc06b99f3c098b905edc6c1f7c4ad1be2a182efefb415ab2dc72d95c2ef31fa0"
Sep 16 17:57:26.153764 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:57:26.153744    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc06b99f3c098b905edc6c1f7c4ad1be2a182efefb415ab2dc72d95c2ef31fa0\": container with ID starting with dc06b99f3c098b905edc6c1f7c4ad1be2a182efefb415ab2dc72d95c2ef31fa0 not found: ID does not exist" containerID="dc06b99f3c098b905edc6c1f7c4ad1be2a182efefb415ab2dc72d95c2ef31fa0"
Sep 16 17:57:26.153811 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:26.153772    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc06b99f3c098b905edc6c1f7c4ad1be2a182efefb415ab2dc72d95c2ef31fa0"} err="failed to get container status \"dc06b99f3c098b905edc6c1f7c4ad1be2a182efefb415ab2dc72d95c2ef31fa0\": rpc error: code = NotFound desc = could not find container \"dc06b99f3c098b905edc6c1f7c4ad1be2a182efefb415ab2dc72d95c2ef31fa0\": container with ID starting with dc06b99f3c098b905edc6c1f7c4ad1be2a182efefb415ab2dc72d95c2ef31fa0 not found: ID does not exist"
Sep 16 17:57:26.191567 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:26.191534    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"]
Sep 16 17:57:26.204153 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:26.204123    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6ks6b"]
Sep 16 17:57:27.080757 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:27.080725    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cdfa8bb-389b-4420-9883-1793ffcfb3a2" path="/var/lib/kubelet/pods/6cdfa8bb-389b-4420-9883-1793ffcfb3a2/volumes"
Sep 16 17:57:33.539298 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:33.539199    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf"
Sep 16 17:57:33.552224 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:33.552195    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf"
Sep 16 17:57:34.188067 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:34.188034    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf"
Sep 16 17:57:39.603715 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:39.603635    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:57:46.463142 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:46.463090    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:57:46.551318 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:46.551280    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:57:52.648967 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:52.648902    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:57:53.066430 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:53.066389    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:57:53.429417 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:53.429328    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:57:53.499761 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:53.499721    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf"]
Sep 16 17:57:53.706656 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:53.706564    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:57:57.636686 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:57.636641    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:57:59.637745 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:59.637716    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg"]
Sep 16 17:57:59.638228 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:59.638209    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cdfa8bb-389b-4420-9883-1793ffcfb3a2" containerName="main"
Sep 16 17:57:59.638277 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:59.638231    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cdfa8bb-389b-4420-9883-1793ffcfb3a2" containerName="main"
Sep 16 17:57:59.638277 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:59.638240    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cdfa8bb-389b-4420-9883-1793ffcfb3a2" containerName="llm-d-routing-sidecar"
Sep 16 17:57:59.638277 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:59.638246    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cdfa8bb-389b-4420-9883-1793ffcfb3a2" containerName="llm-d-routing-sidecar"
Sep 16 17:57:59.638385 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:59.638307    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cdfa8bb-389b-4420-9883-1793ffcfb3a2" containerName="main"
Sep 16 17:57:59.638385 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:59.638314    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cdfa8bb-389b-4420-9883-1793ffcfb3a2" containerName="llm-d-routing-sidecar"
Sep 16 17:57:59.641491 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:59.641470    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg"
Sep 16 17:57:59.644453 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:59.644422    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"openshift-service-ca.crt\""
Sep 16 17:57:59.645602 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:59.645583    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v1-kserve-self-signed-certs\""
Sep 16 17:57:59.645602 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:59.645591    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"kube-root-ca.crt\""
Sep 16 17:57:59.645791 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:59.645599    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"default-dockercfg-4k5vg\""
Sep 16 17:57:59.654632 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:59.654611    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg"]
Sep 16 17:57:59.687942 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:59.687898    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:57:59.821919 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:59.821846    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/107a52f1-5dc6-4b42-ad91-32c350583beb-model-cache\") pod \"test-div-v1-kserve-fb566d667-4c8kg\" (UID: \"107a52f1-5dc6-4b42-ad91-32c350583beb\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg"
Sep 16 17:57:59.822104 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:59.821951    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/107a52f1-5dc6-4b42-ad91-32c350583beb-tls-certs\") pod \"test-div-v1-kserve-fb566d667-4c8kg\" (UID: \"107a52f1-5dc6-4b42-ad91-32c350583beb\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg"
Sep 16 17:57:59.822104 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:59.822026    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/107a52f1-5dc6-4b42-ad91-32c350583beb-dshm\") pod \"test-div-v1-kserve-fb566d667-4c8kg\" (UID: \"107a52f1-5dc6-4b42-ad91-32c350583beb\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg"
Sep 16 17:57:59.822104 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:59.822057    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh6p2\" (UniqueName: \"kubernetes.io/projected/107a52f1-5dc6-4b42-ad91-32c350583beb-kube-api-access-fh6p2\") pod \"test-div-v1-kserve-fb566d667-4c8kg\" (UID: \"107a52f1-5dc6-4b42-ad91-32c350583beb\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg"
Sep 16 17:57:59.822243 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:59.822113    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/107a52f1-5dc6-4b42-ad91-32c350583beb-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-4c8kg\" (UID: \"107a52f1-5dc6-4b42-ad91-32c350583beb\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg"
Sep 16 17:57:59.822243 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:59.822199    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/107a52f1-5dc6-4b42-ad91-32c350583beb-home\") pod \"test-div-v1-kserve-fb566d667-4c8kg\" (UID: \"107a52f1-5dc6-4b42-ad91-32c350583beb\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg"
Sep 16 17:57:59.923440 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:59.923334    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/107a52f1-5dc6-4b42-ad91-32c350583beb-model-cache\") pod \"test-div-v1-kserve-fb566d667-4c8kg\" (UID: \"107a52f1-5dc6-4b42-ad91-32c350583beb\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg"
Sep 16 17:57:59.923440 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:59.923388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/107a52f1-5dc6-4b42-ad91-32c350583beb-tls-certs\") pod \"test-div-v1-kserve-fb566d667-4c8kg\" (UID: \"107a52f1-5dc6-4b42-ad91-32c350583beb\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg"
Sep 16 17:57:59.923440 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:59.923435    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/107a52f1-5dc6-4b42-ad91-32c350583beb-dshm\") pod \"test-div-v1-kserve-fb566d667-4c8kg\" (UID: \"107a52f1-5dc6-4b42-ad91-32c350583beb\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg"
Sep 16 17:57:59.923739 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:59.923465    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh6p2\" (UniqueName: \"kubernetes.io/projected/107a52f1-5dc6-4b42-ad91-32c350583beb-kube-api-access-fh6p2\") pod \"test-div-v1-kserve-fb566d667-4c8kg\" (UID: \"107a52f1-5dc6-4b42-ad91-32c350583beb\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg"
Sep 16 17:57:59.923739 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:59.923530    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/107a52f1-5dc6-4b42-ad91-32c350583beb-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-4c8kg\" (UID: \"107a52f1-5dc6-4b42-ad91-32c350583beb\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg"
Sep 16 17:57:59.923739 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:59.923561    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/107a52f1-5dc6-4b42-ad91-32c350583beb-home\") pod \"test-div-v1-kserve-fb566d667-4c8kg\" (UID: \"107a52f1-5dc6-4b42-ad91-32c350583beb\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg"
Sep 16 17:57:59.923901 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:59.923806    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/107a52f1-5dc6-4b42-ad91-32c350583beb-model-cache\") pod \"test-div-v1-kserve-fb566d667-4c8kg\" (UID: \"107a52f1-5dc6-4b42-ad91-32c350583beb\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg"
Sep 16 17:57:59.923971 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:59.923951    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/107a52f1-5dc6-4b42-ad91-32c350583beb-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-4c8kg\" (UID: \"107a52f1-5dc6-4b42-ad91-32c350583beb\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg"
Sep 16 17:57:59.924061 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:59.924037    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/107a52f1-5dc6-4b42-ad91-32c350583beb-home\") pod \"test-div-v1-kserve-fb566d667-4c8kg\" (UID: \"107a52f1-5dc6-4b42-ad91-32c350583beb\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg"
Sep 16 17:57:59.925798 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:59.925774    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/107a52f1-5dc6-4b42-ad91-32c350583beb-dshm\") pod \"test-div-v1-kserve-fb566d667-4c8kg\" (UID: \"107a52f1-5dc6-4b42-ad91-32c350583beb\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg"
Sep 16 17:57:59.926026 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:59.926009    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/107a52f1-5dc6-4b42-ad91-32c350583beb-tls-certs\") pod \"test-div-v1-kserve-fb566d667-4c8kg\" (UID: \"107a52f1-5dc6-4b42-ad91-32c350583beb\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg"
Sep 16 17:57:59.939230 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:59.939205    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh6p2\" (UniqueName: \"kubernetes.io/projected/107a52f1-5dc6-4b42-ad91-32c350583beb-kube-api-access-fh6p2\") pod \"test-div-v1-kserve-fb566d667-4c8kg\" (UID: \"107a52f1-5dc6-4b42-ad91-32c350583beb\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg"
Sep 16 17:57:59.953030 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:57:59.953009    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg"
Sep 16 17:58:00.229413 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:00.229385    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg"]
Sep 16 17:58:00.230703 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:58:00.230672    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107a52f1_5dc6_4b42_ad91_32c350583beb.slice/crio-54366808e232ba0d49cb40e8ce15cb33b7f0db917c1181b9f8e8648e448b36aa WatchSource:0}: Error finding container 54366808e232ba0d49cb40e8ce15cb33b7f0db917c1181b9f8e8648e448b36aa: Status 404 returned error can't find the container with id 54366808e232ba0d49cb40e8ce15cb33b7f0db917c1181b9f8e8648e448b36aa
Sep 16 17:58:00.315698 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:00.315658    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg" event={"ID":"107a52f1-5dc6-4b42-ad91-32c350583beb","Type":"ContainerStarted","Data":"614691572ccb2d11eeb1d754a1f5bdfbc0a9007520f59ae758cb8828a48f21b3"}
Sep 16 17:58:00.315852 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:00.315724    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg" event={"ID":"107a52f1-5dc6-4b42-ad91-32c350583beb","Type":"ContainerStarted","Data":"54366808e232ba0d49cb40e8ce15cb33b7f0db917c1181b9f8e8648e448b36aa"}
Sep 16 17:58:00.337835 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:00.337777    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg" podStartSLOduration=1.337758507 podStartE2EDuration="1.337758507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:58:00.337298527 +0000 UTC m=+2001.699977313" watchObservedRunningTime="2026-09-16 17:58:00.337758507 +0000 UTC m=+2001.700437345"
Sep 16 17:58:00.954116 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:00.954071    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg"
Sep 16 17:58:06.149385 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:06.149343    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9"]
Sep 16 17:58:06.153069 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:06.153050    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9"
Sep 16 17:58:06.155744 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:06.155718    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v2-kserve-self-signed-certs\""
Sep 16 17:58:06.163240 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:06.163214    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9"]
Sep 16 17:58:06.171368 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:06.171347    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-rhlz9\" (UID: \"7ce1eb56-2a7e-4564-8bf6-1156b9b75512\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9"
Sep 16 17:58:06.171495 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:06.171396    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-home\") pod \"test-div-v2-kserve-5f6fdb6b77-rhlz9\" (UID: \"7ce1eb56-2a7e-4564-8bf6-1156b9b75512\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9"
Sep 16 17:58:06.171495 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:06.171459    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-rhlz9\" (UID: \"7ce1eb56-2a7e-4564-8bf6-1156b9b75512\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9"
Sep 16 17:58:06.171715 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:06.171512    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-rhlz9\" (UID: \"7ce1eb56-2a7e-4564-8bf6-1156b9b75512\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9"
Sep 16 17:58:06.171715 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:06.171573    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-rhlz9\" (UID: \"7ce1eb56-2a7e-4564-8bf6-1156b9b75512\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9"
Sep 16 17:58:06.171715 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:06.171656    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5k7n\" (UniqueName: \"kubernetes.io/projected/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-kube-api-access-f5k7n\") pod \"test-div-v2-kserve-5f6fdb6b77-rhlz9\" (UID: \"7ce1eb56-2a7e-4564-8bf6-1156b9b75512\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9"
Sep 16 17:58:06.246359 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:06.246318    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:58:06.272174 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:06.272137    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-rhlz9\" (UID: \"7ce1eb56-2a7e-4564-8bf6-1156b9b75512\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9"
Sep 16 17:58:06.272364 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:06.272182    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-home\") pod \"test-div-v2-kserve-5f6fdb6b77-rhlz9\" (UID: \"7ce1eb56-2a7e-4564-8bf6-1156b9b75512\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9"
Sep 16 17:58:06.272364 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:06.272204    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-rhlz9\" (UID: \"7ce1eb56-2a7e-4564-8bf6-1156b9b75512\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9"
Sep 16 17:58:06.272364 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:06.272229    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-rhlz9\" (UID: \"7ce1eb56-2a7e-4564-8bf6-1156b9b75512\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9"
Sep 16 17:58:06.272364 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:06.272252    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-rhlz9\" (UID: \"7ce1eb56-2a7e-4564-8bf6-1156b9b75512\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9"
Sep 16 17:58:06.272364 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:06.272286    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5k7n\" (UniqueName: \"kubernetes.io/projected/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-kube-api-access-f5k7n\") pod \"test-div-v2-kserve-5f6fdb6b77-rhlz9\" (UID: \"7ce1eb56-2a7e-4564-8bf6-1156b9b75512\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9"
Sep 16 17:58:06.272649 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:06.272579    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-rhlz9\" (UID: \"7ce1eb56-2a7e-4564-8bf6-1156b9b75512\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9"
Sep 16 17:58:06.272649 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:06.272635    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-home\") pod \"test-div-v2-kserve-5f6fdb6b77-rhlz9\" (UID: \"7ce1eb56-2a7e-4564-8bf6-1156b9b75512\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9"
Sep 16 17:58:06.272732 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:06.272672    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-rhlz9\" (UID: \"7ce1eb56-2a7e-4564-8bf6-1156b9b75512\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9"
Sep 16 17:58:06.274549 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:06.274532    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-rhlz9\" (UID: \"7ce1eb56-2a7e-4564-8bf6-1156b9b75512\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9"
Sep 16 17:58:06.274726 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:06.274709    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-rhlz9\" (UID: \"7ce1eb56-2a7e-4564-8bf6-1156b9b75512\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9"
Sep 16 17:58:06.282129 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:06.282106    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5k7n\" (UniqueName: \"kubernetes.io/projected/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-kube-api-access-f5k7n\") pod \"test-div-v2-kserve-5f6fdb6b77-rhlz9\" (UID: \"7ce1eb56-2a7e-4564-8bf6-1156b9b75512\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9"
Sep 16 17:58:06.465097 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:06.465009    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9"
Sep 16 17:58:06.776525 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:06.776494    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9"]
Sep 16 17:58:06.777604 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:58:06.777576    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ce1eb56_2a7e_4564_8bf6_1156b9b75512.slice/crio-a466c73935ef51cda9b2fe7023ae4584ee533176e2db4dd2eb67557fab84c1ef WatchSource:0}: Error finding container a466c73935ef51cda9b2fe7023ae4584ee533176e2db4dd2eb67557fab84c1ef: Status 404 returned error can't find the container with id a466c73935ef51cda9b2fe7023ae4584ee533176e2db4dd2eb67557fab84c1ef
Sep 16 17:58:07.355509 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:07.355466    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9" event={"ID":"7ce1eb56-2a7e-4564-8bf6-1156b9b75512","Type":"ContainerStarted","Data":"80dbd980f88e0daf878e567b9ee701114045c7af7be7e57c7d3ae80164126a64"}
Sep 16 17:58:07.355935 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:07.355533    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9" event={"ID":"7ce1eb56-2a7e-4564-8bf6-1156b9b75512","Type":"ContainerStarted","Data":"a466c73935ef51cda9b2fe7023ae4584ee533176e2db4dd2eb67557fab84c1ef"}
Sep 16 17:58:07.378189 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:07.378139    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9" podStartSLOduration=1.37812262 podStartE2EDuration="1.37812262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:58:07.376277884 +0000 UTC m=+2008.738956670" watchObservedRunningTime="2026-09-16 17:58:07.37812262 +0000 UTC m=+2008.740801446"
Sep 16 17:58:07.465648 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:07.465612    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9"
Sep 16 17:58:08.518713 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:08.518650    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf" podUID="d3ef9e25-1031-4fae-99db-ecdc76a10483" containerName="main" containerID="cri-o://6997720c2ec5d630b582d7de1a565da65155fe14785dde4d2caceae13ddfb387" gracePeriod=45
Sep 16 17:58:08.784465 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:08.784438    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf"
Sep 16 17:58:08.798681 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:08.798652    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3ef9e25-1031-4fae-99db-ecdc76a10483-dshm\") pod \"d3ef9e25-1031-4fae-99db-ecdc76a10483\" (UID: \"d3ef9e25-1031-4fae-99db-ecdc76a10483\") "
Sep 16 17:58:08.798874 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:08.798696    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3ef9e25-1031-4fae-99db-ecdc76a10483-tmp-dir\") pod \"d3ef9e25-1031-4fae-99db-ecdc76a10483\" (UID: \"d3ef9e25-1031-4fae-99db-ecdc76a10483\") "
Sep 16 17:58:08.798874 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:08.798727    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hf4fm\" (UniqueName: \"kubernetes.io/projected/d3ef9e25-1031-4fae-99db-ecdc76a10483-kube-api-access-hf4fm\") pod \"d3ef9e25-1031-4fae-99db-ecdc76a10483\" (UID: \"d3ef9e25-1031-4fae-99db-ecdc76a10483\") "
Sep 16 17:58:08.798874 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:08.798760    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3ef9e25-1031-4fae-99db-ecdc76a10483-home\") pod \"d3ef9e25-1031-4fae-99db-ecdc76a10483\" (UID: \"d3ef9e25-1031-4fae-99db-ecdc76a10483\") "
Sep 16 17:58:08.798874 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:08.798811    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ef9e25-1031-4fae-99db-ecdc76a10483-tls-certs\") pod \"d3ef9e25-1031-4fae-99db-ecdc76a10483\" (UID: \"d3ef9e25-1031-4fae-99db-ecdc76a10483\") "
Sep 16 17:58:08.798874 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:08.798841    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3ef9e25-1031-4fae-99db-ecdc76a10483-model-cache\") pod \"d3ef9e25-1031-4fae-99db-ecdc76a10483\" (UID: \"d3ef9e25-1031-4fae-99db-ecdc76a10483\") "
Sep 16 17:58:08.799093 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:08.799054    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ef9e25-1031-4fae-99db-ecdc76a10483-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d3ef9e25-1031-4fae-99db-ecdc76a10483" (UID: "d3ef9e25-1031-4fae-99db-ecdc76a10483"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:58:08.799093 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:08.799073    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ef9e25-1031-4fae-99db-ecdc76a10483-home" (OuterVolumeSpecName: "home") pod "d3ef9e25-1031-4fae-99db-ecdc76a10483" (UID: "d3ef9e25-1031-4fae-99db-ecdc76a10483"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:58:08.799267 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:08.799246    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ef9e25-1031-4fae-99db-ecdc76a10483-model-cache" (OuterVolumeSpecName: "model-cache") pod "d3ef9e25-1031-4fae-99db-ecdc76a10483" (UID: "d3ef9e25-1031-4fae-99db-ecdc76a10483"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:58:08.801069 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:08.801041    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ef9e25-1031-4fae-99db-ecdc76a10483-kube-api-access-hf4fm" (OuterVolumeSpecName: "kube-api-access-hf4fm") pod "d3ef9e25-1031-4fae-99db-ecdc76a10483" (UID: "d3ef9e25-1031-4fae-99db-ecdc76a10483"). InnerVolumeSpecName "kube-api-access-hf4fm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:58:08.801171 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:08.801062    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3ef9e25-1031-4fae-99db-ecdc76a10483-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d3ef9e25-1031-4fae-99db-ecdc76a10483" (UID: "d3ef9e25-1031-4fae-99db-ecdc76a10483"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:58:08.801171 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:08.801090    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ef9e25-1031-4fae-99db-ecdc76a10483-dshm" (OuterVolumeSpecName: "dshm") pod "d3ef9e25-1031-4fae-99db-ecdc76a10483" (UID: "d3ef9e25-1031-4fae-99db-ecdc76a10483"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:58:08.899644 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:08.899589    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3ef9e25-1031-4fae-99db-ecdc76a10483-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:58:08.899644 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:08.899637    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf4fm\" (UniqueName: \"kubernetes.io/projected/d3ef9e25-1031-4fae-99db-ecdc76a10483-kube-api-access-hf4fm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:58:08.899644 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:08.899651    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3ef9e25-1031-4fae-99db-ecdc76a10483-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:58:08.899934 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:08.899663    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ef9e25-1031-4fae-99db-ecdc76a10483-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:58:08.899934 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:08.899674    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3ef9e25-1031-4fae-99db-ecdc76a10483-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:58:08.899934 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:08.899685    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3ef9e25-1031-4fae-99db-ecdc76a10483-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:58:09.367190 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:09.367102    2577 generic.go:356] "Generic (PLEG): container finished" podID="d3ef9e25-1031-4fae-99db-ecdc76a10483" containerID="6997720c2ec5d630b582d7de1a565da65155fe14785dde4d2caceae13ddfb387" exitCode=0
Sep 16 17:58:09.367190 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:09.367143    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf"
Sep 16 17:58:09.367423 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:09.367179    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf" event={"ID":"d3ef9e25-1031-4fae-99db-ecdc76a10483","Type":"ContainerDied","Data":"6997720c2ec5d630b582d7de1a565da65155fe14785dde4d2caceae13ddfb387"}
Sep 16 17:58:09.367423 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:09.367252    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf" event={"ID":"d3ef9e25-1031-4fae-99db-ecdc76a10483","Type":"ContainerDied","Data":"12a7ab49ecaf64442a08a870108fe2f59feee3aac580dd00125f4804f62cb3fb"}
Sep 16 17:58:09.367423 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:09.367283    2577 scope.go:117] "RemoveContainer" containerID="6997720c2ec5d630b582d7de1a565da65155fe14785dde4d2caceae13ddfb387"
Sep 16 17:58:09.377714 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:09.377691    2577 scope.go:117] "RemoveContainer" containerID="6997720c2ec5d630b582d7de1a565da65155fe14785dde4d2caceae13ddfb387"
Sep 16 17:58:09.378047 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:58:09.378021    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6997720c2ec5d630b582d7de1a565da65155fe14785dde4d2caceae13ddfb387\": container with ID starting with 6997720c2ec5d630b582d7de1a565da65155fe14785dde4d2caceae13ddfb387 not found: ID does not exist" containerID="6997720c2ec5d630b582d7de1a565da65155fe14785dde4d2caceae13ddfb387"
Sep 16 17:58:09.378123 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:09.378049    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6997720c2ec5d630b582d7de1a565da65155fe14785dde4d2caceae13ddfb387"} err="failed to get container status \"6997720c2ec5d630b582d7de1a565da65155fe14785dde4d2caceae13ddfb387\": rpc error: code = NotFound desc = could not find container \"6997720c2ec5d630b582d7de1a565da65155fe14785dde4d2caceae13ddfb387\": container with ID starting with 6997720c2ec5d630b582d7de1a565da65155fe14785dde4d2caceae13ddfb387 not found: ID does not exist"
Sep 16 17:58:09.393251 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:09.393224    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf"]
Sep 16 17:58:09.397052 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:09.397032    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-b2cxf"]
Sep 16 17:58:09.954201 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:09.954160    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg"
Sep 16 17:58:09.966729 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:09.966699    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg"
Sep 16 17:58:10.384608 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:10.384575    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg"
Sep 16 17:58:11.081264 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:11.081230    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3ef9e25-1031-4fae-99db-ecdc76a10483" path="/var/lib/kubelet/pods/d3ef9e25-1031-4fae-99db-ecdc76a10483/volumes"
Sep 16 17:58:13.464460 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:13.464425    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:58:13.584836 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:13.584796    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:58:16.465837 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:16.465801    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9"
Sep 16 17:58:16.478200 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:16.478176    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9"
Sep 16 17:58:17.423006 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:17.422973    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9"
Sep 16 17:58:20.685224 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:20.685187    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:58:26.846511 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:26.846474    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:58:26.946682 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:26.946647    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:58:35.694963 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:35.694901    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:58:42.432675 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:42.432638    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:58:42.565450 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:42.565414    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:58:46.434244 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:46.434209    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"]
Sep 16 17:58:46.434605 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:46.434593    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3ef9e25-1031-4fae-99db-ecdc76a10483" containerName="main"
Sep 16 17:58:46.434652 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:46.434604    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3ef9e25-1031-4fae-99db-ecdc76a10483" containerName="main"
Sep 16 17:58:46.434687 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:46.434676    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3ef9e25-1031-4fae-99db-ecdc76a10483" containerName="main"
Sep 16 17:58:46.437195 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:46.437176    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"
Sep 16 17:58:46.446301 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:46.446277    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-s7h7d\""
Sep 16 17:58:46.446428 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:46.446298    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 16 17:58:46.446428 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:46.446321    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 16 17:58:46.446428 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:46.446325    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-g67xf\""
Sep 16 17:58:46.446428 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:46.446298    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 16 17:58:46.483193 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:46.483165    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"]
Sep 16 17:58:46.518319 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:46.518277    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:58:46.520289 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:46.520265    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74f7cf46-5bf1-4093-931c-ff64378b3814-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6\" (UID: \"74f7cf46-5bf1-4093-931c-ff64378b3814\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"
Sep 16 17:58:46.520373 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:46.520303    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74f7cf46-5bf1-4093-931c-ff64378b3814-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6\" (UID: \"74f7cf46-5bf1-4093-931c-ff64378b3814\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"
Sep 16 17:58:46.520373 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:46.520319    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74f7cf46-5bf1-4093-931c-ff64378b3814-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6\" (UID: \"74f7cf46-5bf1-4093-931c-ff64378b3814\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"
Sep 16 17:58:46.520460 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:46.520394    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74f7cf46-5bf1-4093-931c-ff64378b3814-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6\" (UID: \"74f7cf46-5bf1-4093-931c-ff64378b3814\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"
Sep 16 17:58:46.520460 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:46.520445    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7gw4\" (UniqueName: \"kubernetes.io/projected/74f7cf46-5bf1-4093-931c-ff64378b3814-kube-api-access-v7gw4\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6\" (UID: \"74f7cf46-5bf1-4093-931c-ff64378b3814\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"
Sep 16 17:58:46.520588 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:46.520466    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74f7cf46-5bf1-4093-931c-ff64378b3814-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6\" (UID: \"74f7cf46-5bf1-4093-931c-ff64378b3814\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"
Sep 16 17:58:46.621735 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:46.621692    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7gw4\" (UniqueName: \"kubernetes.io/projected/74f7cf46-5bf1-4093-931c-ff64378b3814-kube-api-access-v7gw4\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6\" (UID: \"74f7cf46-5bf1-4093-931c-ff64378b3814\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"
Sep 16 17:58:46.621984 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:46.621746    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74f7cf46-5bf1-4093-931c-ff64378b3814-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6\" (UID: \"74f7cf46-5bf1-4093-931c-ff64378b3814\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"
Sep 16 17:58:46.621984 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:46.621841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74f7cf46-5bf1-4093-931c-ff64378b3814-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6\" (UID: \"74f7cf46-5bf1-4093-931c-ff64378b3814\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"
Sep 16 17:58:46.621984 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:46.621909    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74f7cf46-5bf1-4093-931c-ff64378b3814-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6\" (UID: \"74f7cf46-5bf1-4093-931c-ff64378b3814\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"
Sep 16 17:58:46.621984 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:46.621931    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74f7cf46-5bf1-4093-931c-ff64378b3814-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6\" (UID: \"74f7cf46-5bf1-4093-931c-ff64378b3814\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"
Sep 16 17:58:46.621984 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:46.621979    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74f7cf46-5bf1-4093-931c-ff64378b3814-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6\" (UID: \"74f7cf46-5bf1-4093-931c-ff64378b3814\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"
Sep 16 17:58:46.622273 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:46.622247    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74f7cf46-5bf1-4093-931c-ff64378b3814-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6\" (UID: \"74f7cf46-5bf1-4093-931c-ff64378b3814\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"
Sep 16 17:58:46.622335 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:46.622306    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74f7cf46-5bf1-4093-931c-ff64378b3814-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6\" (UID: \"74f7cf46-5bf1-4093-931c-ff64378b3814\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"
Sep 16 17:58:46.622397 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:46.622380    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74f7cf46-5bf1-4093-931c-ff64378b3814-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6\" (UID: \"74f7cf46-5bf1-4093-931c-ff64378b3814\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"
Sep 16 17:58:46.624175 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:46.624150    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74f7cf46-5bf1-4093-931c-ff64378b3814-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6\" (UID: \"74f7cf46-5bf1-4093-931c-ff64378b3814\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"
Sep 16 17:58:46.624347 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:46.624332    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74f7cf46-5bf1-4093-931c-ff64378b3814-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6\" (UID: \"74f7cf46-5bf1-4093-931c-ff64378b3814\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"
Sep 16 17:58:46.636890 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:46.636835    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7gw4\" (UniqueName: \"kubernetes.io/projected/74f7cf46-5bf1-4093-931c-ff64378b3814-kube-api-access-v7gw4\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6\" (UID: \"74f7cf46-5bf1-4093-931c-ff64378b3814\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"
Sep 16 17:58:46.746114 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:46.746019    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"
Sep 16 17:58:46.954919 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:46.954857    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:58:47.094584 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:58:47.094552    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f7cf46_5bf1_4093_931c_ff64378b3814.slice/crio-6d265e987d4e14223cbd626b8200b4541395347f4e77ca29b501e24165767140 WatchSource:0}: Error finding container 6d265e987d4e14223cbd626b8200b4541395347f4e77ca29b501e24165767140: Status 404 returned error can't find the container with id 6d265e987d4e14223cbd626b8200b4541395347f4e77ca29b501e24165767140
Sep 16 17:58:47.102348 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:47.102323    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"]
Sep 16 17:58:47.578577 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:47.578530    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6" event={"ID":"74f7cf46-5bf1-4093-931c-ff64378b3814","Type":"ContainerStarted","Data":"198102b4bccd946e7fea11c5c2f86220424541cc4ea8c95526819613fd2dce92"}
Sep 16 17:58:47.578993 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:47.578595    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6" event={"ID":"74f7cf46-5bf1-4093-931c-ff64378b3814","Type":"ContainerStarted","Data":"6d265e987d4e14223cbd626b8200b4541395347f4e77ca29b501e24165767140"}
Sep 16 17:58:47.578993 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:47.578627    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"
Sep 16 17:58:48.588517 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:48.588480    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6" event={"ID":"74f7cf46-5bf1-4093-931c-ff64378b3814","Type":"ContainerStarted","Data":"84e59760b4b18715f60ca5a37017d117637cfd2b06a88f5a7dff4d37bf2faf48"}
Sep 16 17:58:48.637906 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:48.637838    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6" podStartSLOduration=2.637820272 podStartE2EDuration="2.637820272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:58:48.636994058 +0000 UTC m=+2049.999672845" watchObservedRunningTime="2026-09-16 17:58:48.637820272 +0000 UTC m=+2050.000499057"
Sep 16 17:58:48.746602 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:48.746560    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"
Sep 16 17:58:56.746647 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:56.746609    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"
Sep 16 17:58:56.759183 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:56.759156    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"
Sep 16 17:58:57.649703 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:57.649662    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"
Sep 16 17:58:57.650789 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:58:57.650768    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"
Sep 16 17:59:06.385722 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:06.385638    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:59:11.193697 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:11.193653    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:59:11.339315 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:11.339277    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:59:11.497332 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:11.497248    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:59:11.901344 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:11.901307    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:59:12.054116 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:12.054082    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:59:12.721945 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:12.721911    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9"]
Sep 16 17:59:12.759789 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:12.759754    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:59:13.474561 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:13.474523    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg"]
Sep 16 17:59:13.507200 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:13.507167    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:59:16.263505 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:16.263475    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:59:18.733002 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:18.732972    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:59:20.101556 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:20.101512    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:59:23.368109 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:23.368076    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94"]
Sep 16 17:59:23.372075 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:23.372053    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94"
Sep 16 17:59:23.376175 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:23.376149    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"default-dockercfg-2zlh7\""
Sep 16 17:59:23.376309 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:23.376176    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"openshift-service-ca.crt\""
Sep 16 17:59:23.376309 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:23.376189    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"kube-root-ca.crt\""
Sep 16 17:59:23.376309 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:23.376158    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v2-kserve-self-signed-certs\""
Sep 16 17:59:23.401105 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:23.401070    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94"]
Sep 16 17:59:23.454585 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:23.454553    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7xkb\" (UniqueName: \"kubernetes.io/projected/125127f8-862a-4225-b042-16f0d13c377c-kube-api-access-q7xkb\") pod \"stop-v2-kserve-7f68747c45-dcz94\" (UID: \"125127f8-862a-4225-b042-16f0d13c377c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94"
Sep 16 17:59:23.454779 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:23.454595    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/125127f8-862a-4225-b042-16f0d13c377c-tls-certs\") pod \"stop-v2-kserve-7f68747c45-dcz94\" (UID: \"125127f8-862a-4225-b042-16f0d13c377c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94"
Sep 16 17:59:23.454779 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:23.454643    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/125127f8-862a-4225-b042-16f0d13c377c-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-dcz94\" (UID: \"125127f8-862a-4225-b042-16f0d13c377c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94"
Sep 16 17:59:23.454779 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:23.454694    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/125127f8-862a-4225-b042-16f0d13c377c-model-cache\") pod \"stop-v2-kserve-7f68747c45-dcz94\" (UID: \"125127f8-862a-4225-b042-16f0d13c377c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94"
Sep 16 17:59:23.454779 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:23.454764    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/125127f8-862a-4225-b042-16f0d13c377c-home\") pod \"stop-v2-kserve-7f68747c45-dcz94\" (UID: \"125127f8-862a-4225-b042-16f0d13c377c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94"
Sep 16 17:59:23.454998 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:23.454788    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/125127f8-862a-4225-b042-16f0d13c377c-dshm\") pod \"stop-v2-kserve-7f68747c45-dcz94\" (UID: \"125127f8-862a-4225-b042-16f0d13c377c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94"
Sep 16 17:59:23.480230 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:23.480191    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:59:23.556299 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:23.556259    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7xkb\" (UniqueName: \"kubernetes.io/projected/125127f8-862a-4225-b042-16f0d13c377c-kube-api-access-q7xkb\") pod \"stop-v2-kserve-7f68747c45-dcz94\" (UID: \"125127f8-862a-4225-b042-16f0d13c377c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94"
Sep 16 17:59:23.556500 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:23.556312    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/125127f8-862a-4225-b042-16f0d13c377c-tls-certs\") pod \"stop-v2-kserve-7f68747c45-dcz94\" (UID: \"125127f8-862a-4225-b042-16f0d13c377c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94"
Sep 16 17:59:23.556500 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:23.556336    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/125127f8-862a-4225-b042-16f0d13c377c-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-dcz94\" (UID: \"125127f8-862a-4225-b042-16f0d13c377c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94"
Sep 16 17:59:23.556500 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:23.556358    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/125127f8-862a-4225-b042-16f0d13c377c-model-cache\") pod \"stop-v2-kserve-7f68747c45-dcz94\" (UID: \"125127f8-862a-4225-b042-16f0d13c377c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94"
Sep 16 17:59:23.556500 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:23.556406    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/125127f8-862a-4225-b042-16f0d13c377c-home\") pod \"stop-v2-kserve-7f68747c45-dcz94\" (UID: \"125127f8-862a-4225-b042-16f0d13c377c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94"
Sep 16 17:59:23.556702 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:23.556541    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/125127f8-862a-4225-b042-16f0d13c377c-dshm\") pod \"stop-v2-kserve-7f68747c45-dcz94\" (UID: \"125127f8-862a-4225-b042-16f0d13c377c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94"
Sep 16 17:59:23.556828 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:23.556808    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/125127f8-862a-4225-b042-16f0d13c377c-home\") pod \"stop-v2-kserve-7f68747c45-dcz94\" (UID: \"125127f8-862a-4225-b042-16f0d13c377c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94"
Sep 16 17:59:23.556933 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:23.556840    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/125127f8-862a-4225-b042-16f0d13c377c-model-cache\") pod \"stop-v2-kserve-7f68747c45-dcz94\" (UID: \"125127f8-862a-4225-b042-16f0d13c377c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94"
Sep 16 17:59:23.556933 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:23.556859    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/125127f8-862a-4225-b042-16f0d13c377c-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-dcz94\" (UID: \"125127f8-862a-4225-b042-16f0d13c377c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94"
Sep 16 17:59:23.558815 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:23.558793    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/125127f8-862a-4225-b042-16f0d13c377c-dshm\") pod \"stop-v2-kserve-7f68747c45-dcz94\" (UID: \"125127f8-862a-4225-b042-16f0d13c377c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94"
Sep 16 17:59:23.559108 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:23.559090    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/125127f8-862a-4225-b042-16f0d13c377c-tls-certs\") pod \"stop-v2-kserve-7f68747c45-dcz94\" (UID: \"125127f8-862a-4225-b042-16f0d13c377c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94"
Sep 16 17:59:23.567267 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:23.567244    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7xkb\" (UniqueName: \"kubernetes.io/projected/125127f8-862a-4225-b042-16f0d13c377c-kube-api-access-q7xkb\") pod \"stop-v2-kserve-7f68747c45-dcz94\" (UID: \"125127f8-862a-4225-b042-16f0d13c377c\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94"
Sep 16 17:59:23.683846 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:23.683757    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94"
Sep 16 17:59:23.959970 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:23.959936    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94"]
Sep 16 17:59:23.961493 ip-10-0-136-122 kubenswrapper[2577]: W0916 17:59:23.961469    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod125127f8_862a_4225_b042_16f0d13c377c.slice/crio-f63fdd7b2cdb91206d08a1242912109b0cf1402f832140e305bd872ff3683ee6 WatchSource:0}: Error finding container f63fdd7b2cdb91206d08a1242912109b0cf1402f832140e305bd872ff3683ee6: Status 404 returned error can't find the container with id f63fdd7b2cdb91206d08a1242912109b0cf1402f832140e305bd872ff3683ee6
Sep 16 17:59:24.786436 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:24.786398    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94" event={"ID":"125127f8-862a-4225-b042-16f0d13c377c","Type":"ContainerStarted","Data":"55ee61bf5d923b3fd98ad5119f8315cbfbead04941dd95c44a6809c02fca24c2"}
Sep 16 17:59:24.786840 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:24.786483    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94" event={"ID":"125127f8-862a-4225-b042-16f0d13c377c","Type":"ContainerStarted","Data":"f63fdd7b2cdb91206d08a1242912109b0cf1402f832140e305bd872ff3683ee6"}
Sep 16 17:59:24.843569 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:24.843510    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94" podStartSLOduration=1.843494806 podStartE2EDuration="1.843494806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:59:24.841472966 +0000 UTC m=+2086.204151752" watchObservedRunningTime="2026-09-16 17:59:24.843494806 +0000 UTC m=+2086.206173591"
Sep 16 17:59:25.684178 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:25.684144    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94"
Sep 16 17:59:27.123923 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:27.123855    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:59:27.245449 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:27.245401    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:59:27.740593 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:27.740553    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9" podUID="7ce1eb56-2a7e-4564-8bf6-1156b9b75512" containerName="main" containerID="cri-o://80dbd980f88e0daf878e567b9ee701114045c7af7be7e57c7d3ae80164126a64" gracePeriod=45
Sep 16 17:59:27.998210 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:27.998153    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9"
Sep 16 17:59:28.096796 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.096760    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-model-cache\") pod \"7ce1eb56-2a7e-4564-8bf6-1156b9b75512\" (UID: \"7ce1eb56-2a7e-4564-8bf6-1156b9b75512\") "
Sep 16 17:59:28.096796 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.096796    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5k7n\" (UniqueName: \"kubernetes.io/projected/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-kube-api-access-f5k7n\") pod \"7ce1eb56-2a7e-4564-8bf6-1156b9b75512\" (UID: \"7ce1eb56-2a7e-4564-8bf6-1156b9b75512\") "
Sep 16 17:59:28.097062 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.096835    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-tmp-dir\") pod \"7ce1eb56-2a7e-4564-8bf6-1156b9b75512\" (UID: \"7ce1eb56-2a7e-4564-8bf6-1156b9b75512\") "
Sep 16 17:59:28.097062 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.096901    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-dshm\") pod \"7ce1eb56-2a7e-4564-8bf6-1156b9b75512\" (UID: \"7ce1eb56-2a7e-4564-8bf6-1156b9b75512\") "
Sep 16 17:59:28.097062 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.096939    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-tls-certs\") pod \"7ce1eb56-2a7e-4564-8bf6-1156b9b75512\" (UID: \"7ce1eb56-2a7e-4564-8bf6-1156b9b75512\") "
Sep 16 17:59:28.097062 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.096969    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-home\") pod \"7ce1eb56-2a7e-4564-8bf6-1156b9b75512\" (UID: \"7ce1eb56-2a7e-4564-8bf6-1156b9b75512\") "
Sep 16 17:59:28.097269 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.097090    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-model-cache" (OuterVolumeSpecName: "model-cache") pod "7ce1eb56-2a7e-4564-8bf6-1156b9b75512" (UID: "7ce1eb56-2a7e-4564-8bf6-1156b9b75512"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:59:28.097269 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.097098    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7ce1eb56-2a7e-4564-8bf6-1156b9b75512" (UID: "7ce1eb56-2a7e-4564-8bf6-1156b9b75512"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:59:28.097372 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.097262    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:59:28.097372 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.097287    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:59:28.097372 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.097299    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-home" (OuterVolumeSpecName: "home") pod "7ce1eb56-2a7e-4564-8bf6-1156b9b75512" (UID: "7ce1eb56-2a7e-4564-8bf6-1156b9b75512"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:59:28.099189 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.099166    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-dshm" (OuterVolumeSpecName: "dshm") pod "7ce1eb56-2a7e-4564-8bf6-1156b9b75512" (UID: "7ce1eb56-2a7e-4564-8bf6-1156b9b75512"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:59:28.099286 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.099221    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-kube-api-access-f5k7n" (OuterVolumeSpecName: "kube-api-access-f5k7n") pod "7ce1eb56-2a7e-4564-8bf6-1156b9b75512" (UID: "7ce1eb56-2a7e-4564-8bf6-1156b9b75512"). InnerVolumeSpecName "kube-api-access-f5k7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:59:28.099286 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.099221    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7ce1eb56-2a7e-4564-8bf6-1156b9b75512" (UID: "7ce1eb56-2a7e-4564-8bf6-1156b9b75512"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:59:28.198503 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.198467    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:59:28.198503 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.198504    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:59:28.198503 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.198514    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:59:28.199033 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.198524    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5k7n\" (UniqueName: \"kubernetes.io/projected/7ce1eb56-2a7e-4564-8bf6-1156b9b75512-kube-api-access-f5k7n\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:59:28.493977 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.493932    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg" podUID="107a52f1-5dc6-4b42-ad91-32c350583beb" containerName="main" containerID="cri-o://614691572ccb2d11eeb1d754a1f5bdfbc0a9007520f59ae758cb8828a48f21b3" gracePeriod=45
Sep 16 17:59:28.729611 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.729586    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg"
Sep 16 17:59:28.803910 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.803854    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/107a52f1-5dc6-4b42-ad91-32c350583beb-home\") pod \"107a52f1-5dc6-4b42-ad91-32c350583beb\" (UID: \"107a52f1-5dc6-4b42-ad91-32c350583beb\") "
Sep 16 17:59:28.804099 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.803932    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/107a52f1-5dc6-4b42-ad91-32c350583beb-dshm\") pod \"107a52f1-5dc6-4b42-ad91-32c350583beb\" (UID: \"107a52f1-5dc6-4b42-ad91-32c350583beb\") "
Sep 16 17:59:28.804099 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.803995    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/107a52f1-5dc6-4b42-ad91-32c350583beb-tmp-dir\") pod \"107a52f1-5dc6-4b42-ad91-32c350583beb\" (UID: \"107a52f1-5dc6-4b42-ad91-32c350583beb\") "
Sep 16 17:59:28.804099 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.804023    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/107a52f1-5dc6-4b42-ad91-32c350583beb-model-cache\") pod \"107a52f1-5dc6-4b42-ad91-32c350583beb\" (UID: \"107a52f1-5dc6-4b42-ad91-32c350583beb\") "
Sep 16 17:59:28.804099 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.804053    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/107a52f1-5dc6-4b42-ad91-32c350583beb-tls-certs\") pod \"107a52f1-5dc6-4b42-ad91-32c350583beb\" (UID: \"107a52f1-5dc6-4b42-ad91-32c350583beb\") "
Sep 16 17:59:28.804486 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.804131    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fh6p2\" (UniqueName: \"kubernetes.io/projected/107a52f1-5dc6-4b42-ad91-32c350583beb-kube-api-access-fh6p2\") pod \"107a52f1-5dc6-4b42-ad91-32c350583beb\" (UID: \"107a52f1-5dc6-4b42-ad91-32c350583beb\") "
Sep 16 17:59:28.804486 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.804290    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107a52f1-5dc6-4b42-ad91-32c350583beb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "107a52f1-5dc6-4b42-ad91-32c350583beb" (UID: "107a52f1-5dc6-4b42-ad91-32c350583beb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:59:28.804486 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.804378    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/107a52f1-5dc6-4b42-ad91-32c350583beb-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:59:28.804486 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.804480    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107a52f1-5dc6-4b42-ad91-32c350583beb-model-cache" (OuterVolumeSpecName: "model-cache") pod "107a52f1-5dc6-4b42-ad91-32c350583beb" (UID: "107a52f1-5dc6-4b42-ad91-32c350583beb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:59:28.804797 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.804535    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107a52f1-5dc6-4b42-ad91-32c350583beb-home" (OuterVolumeSpecName: "home") pod "107a52f1-5dc6-4b42-ad91-32c350583beb" (UID: "107a52f1-5dc6-4b42-ad91-32c350583beb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:59:28.806890 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.806816    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107a52f1-5dc6-4b42-ad91-32c350583beb-kube-api-access-fh6p2" (OuterVolumeSpecName: "kube-api-access-fh6p2") pod "107a52f1-5dc6-4b42-ad91-32c350583beb" (UID: "107a52f1-5dc6-4b42-ad91-32c350583beb"). InnerVolumeSpecName "kube-api-access-fh6p2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:59:28.807188 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.807165    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/107a52f1-5dc6-4b42-ad91-32c350583beb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "107a52f1-5dc6-4b42-ad91-32c350583beb" (UID: "107a52f1-5dc6-4b42-ad91-32c350583beb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:59:28.807383 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.807348    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107a52f1-5dc6-4b42-ad91-32c350583beb-dshm" (OuterVolumeSpecName: "dshm") pod "107a52f1-5dc6-4b42-ad91-32c350583beb" (UID: "107a52f1-5dc6-4b42-ad91-32c350583beb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:59:28.811444 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.811398    2577 generic.go:356] "Generic (PLEG): container finished" podID="7ce1eb56-2a7e-4564-8bf6-1156b9b75512" containerID="80dbd980f88e0daf878e567b9ee701114045c7af7be7e57c7d3ae80164126a64" exitCode=0
Sep 16 17:59:28.811651 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.811487    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9"
Sep 16 17:59:28.811651 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.811528    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9" event={"ID":"7ce1eb56-2a7e-4564-8bf6-1156b9b75512","Type":"ContainerDied","Data":"80dbd980f88e0daf878e567b9ee701114045c7af7be7e57c7d3ae80164126a64"}
Sep 16 17:59:28.811651 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.811580    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9" event={"ID":"7ce1eb56-2a7e-4564-8bf6-1156b9b75512","Type":"ContainerDied","Data":"a466c73935ef51cda9b2fe7023ae4584ee533176e2db4dd2eb67557fab84c1ef"}
Sep 16 17:59:28.811651 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.811611    2577 scope.go:117] "RemoveContainer" containerID="80dbd980f88e0daf878e567b9ee701114045c7af7be7e57c7d3ae80164126a64"
Sep 16 17:59:28.813603 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.813457    2577 generic.go:356] "Generic (PLEG): container finished" podID="107a52f1-5dc6-4b42-ad91-32c350583beb" containerID="614691572ccb2d11eeb1d754a1f5bdfbc0a9007520f59ae758cb8828a48f21b3" exitCode=0
Sep 16 17:59:28.813603 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.813507    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg" event={"ID":"107a52f1-5dc6-4b42-ad91-32c350583beb","Type":"ContainerDied","Data":"614691572ccb2d11eeb1d754a1f5bdfbc0a9007520f59ae758cb8828a48f21b3"}
Sep 16 17:59:28.813603 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.813578    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg" event={"ID":"107a52f1-5dc6-4b42-ad91-32c350583beb","Type":"ContainerDied","Data":"54366808e232ba0d49cb40e8ce15cb33b7f0db917c1181b9f8e8648e448b36aa"}
Sep 16 17:59:28.813603 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.813510    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg"
Sep 16 17:59:28.830356 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.830333    2577 scope.go:117] "RemoveContainer" containerID="80dbd980f88e0daf878e567b9ee701114045c7af7be7e57c7d3ae80164126a64"
Sep 16 17:59:28.830698 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:59:28.830672    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80dbd980f88e0daf878e567b9ee701114045c7af7be7e57c7d3ae80164126a64\": container with ID starting with 80dbd980f88e0daf878e567b9ee701114045c7af7be7e57c7d3ae80164126a64 not found: ID does not exist" containerID="80dbd980f88e0daf878e567b9ee701114045c7af7be7e57c7d3ae80164126a64"
Sep 16 17:59:28.830770 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.830711    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80dbd980f88e0daf878e567b9ee701114045c7af7be7e57c7d3ae80164126a64"} err="failed to get container status \"80dbd980f88e0daf878e567b9ee701114045c7af7be7e57c7d3ae80164126a64\": rpc error: code = NotFound desc = could not find container \"80dbd980f88e0daf878e567b9ee701114045c7af7be7e57c7d3ae80164126a64\": container with ID starting with 80dbd980f88e0daf878e567b9ee701114045c7af7be7e57c7d3ae80164126a64 not found: ID does not exist"
Sep 16 17:59:28.830770 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.830738    2577 scope.go:117] "RemoveContainer" containerID="614691572ccb2d11eeb1d754a1f5bdfbc0a9007520f59ae758cb8828a48f21b3"
Sep 16 17:59:28.841610 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.841588    2577 scope.go:117] "RemoveContainer" containerID="614691572ccb2d11eeb1d754a1f5bdfbc0a9007520f59ae758cb8828a48f21b3"
Sep 16 17:59:28.841998 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:59:28.841970    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"614691572ccb2d11eeb1d754a1f5bdfbc0a9007520f59ae758cb8828a48f21b3\": container with ID starting with 614691572ccb2d11eeb1d754a1f5bdfbc0a9007520f59ae758cb8828a48f21b3 not found: ID does not exist" containerID="614691572ccb2d11eeb1d754a1f5bdfbc0a9007520f59ae758cb8828a48f21b3"
Sep 16 17:59:28.842093 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.842003    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"614691572ccb2d11eeb1d754a1f5bdfbc0a9007520f59ae758cb8828a48f21b3"} err="failed to get container status \"614691572ccb2d11eeb1d754a1f5bdfbc0a9007520f59ae758cb8828a48f21b3\": rpc error: code = NotFound desc = could not find container \"614691572ccb2d11eeb1d754a1f5bdfbc0a9007520f59ae758cb8828a48f21b3\": container with ID starting with 614691572ccb2d11eeb1d754a1f5bdfbc0a9007520f59ae758cb8828a48f21b3 not found: ID does not exist"
Sep 16 17:59:28.869529 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.869501    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9"]
Sep 16 17:59:28.889173 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.889142    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-rhlz9"]
Sep 16 17:59:28.905249 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.905223    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh6p2\" (UniqueName: \"kubernetes.io/projected/107a52f1-5dc6-4b42-ad91-32c350583beb-kube-api-access-fh6p2\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:59:28.905249 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.905245    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/107a52f1-5dc6-4b42-ad91-32c350583beb-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:59:28.905392 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.905254    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/107a52f1-5dc6-4b42-ad91-32c350583beb-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:59:28.905392 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.905263    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/107a52f1-5dc6-4b42-ad91-32c350583beb-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:59:28.905392 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.905272    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/107a52f1-5dc6-4b42-ad91-32c350583beb-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:59:28.929383 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.929336    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg"]
Sep 16 17:59:28.941664 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:28.941632    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-4c8kg"]
Sep 16 17:59:29.087035 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:29.087000    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="107a52f1-5dc6-4b42-ad91-32c350583beb" path="/var/lib/kubelet/pods/107a52f1-5dc6-4b42-ad91-32c350583beb/volumes"
Sep 16 17:59:29.087428 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:29.087415    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ce1eb56-2a7e-4564-8bf6-1156b9b75512" path="/var/lib/kubelet/pods/7ce1eb56-2a7e-4564-8bf6-1156b9b75512/volumes"
Sep 16 17:59:33.684147 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:33.684113    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94"
Sep 16 17:59:33.696452 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:33.696426    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94"
Sep 16 17:59:33.853633 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:33.853604    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94"
Sep 16 17:59:51.802503 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:51.802451    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:59:51.827741 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:51.827706    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"]
Sep 16 17:59:51.837828 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:59:51.837784    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 17:59:51.837828 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:59:51.837828 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 17:59:51.837828 ip-10-0-136-122 kubenswrapper[2577]:  > containerID="84e59760b4b18715f60ca5a37017d117637cfd2b06a88f5a7dff4d37bf2faf48" cmd=["/bin/sleep","15"]
Sep 16 17:59:51.838106 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:51.837847    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6" message="" err=<
Sep 16 17:59:51.838106 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:59:51.838106 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 17:59:51.838106 ip-10-0-136-122 kubenswrapper[2577]:  >
Sep 16 17:59:51.838106 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:59:51.837905    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 17:59:51.838106 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:59:51.838106 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 17:59:51.838106 ip-10-0-136-122 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6" podUID="74f7cf46-5bf1-4093-931c-ff64378b3814" containerName="main" containerID="cri-o://84e59760b4b18715f60ca5a37017d117637cfd2b06a88f5a7dff4d37bf2faf48"
Sep 16 17:59:51.838106 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:51.837956    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6" podUID="74f7cf46-5bf1-4093-931c-ff64378b3814" containerName="main" containerID="cri-o://84e59760b4b18715f60ca5a37017d117637cfd2b06a88f5a7dff4d37bf2faf48" gracePeriod=60
Sep 16 17:59:51.902645 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:51.902616    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:59:52.053596 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.053566    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6" podUID="74f7cf46-5bf1-4093-931c-ff64378b3814" containerName="llm-d-routing-sidecar" containerID="cri-o://198102b4bccd946e7fea11c5c2f86220424541cc4ea8c95526819613fd2dce92" gracePeriod=60
Sep 16 17:59:52.309273 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.309213    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"
Sep 16 17:59:52.404948 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.404917    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74f7cf46-5bf1-4093-931c-ff64378b3814-home\") pod \"74f7cf46-5bf1-4093-931c-ff64378b3814\" (UID: \"74f7cf46-5bf1-4093-931c-ff64378b3814\") "
Sep 16 17:59:52.404948 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.404952    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7gw4\" (UniqueName: \"kubernetes.io/projected/74f7cf46-5bf1-4093-931c-ff64378b3814-kube-api-access-v7gw4\") pod \"74f7cf46-5bf1-4093-931c-ff64378b3814\" (UID: \"74f7cf46-5bf1-4093-931c-ff64378b3814\") "
Sep 16 17:59:52.405169 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.404989    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74f7cf46-5bf1-4093-931c-ff64378b3814-model-cache\") pod \"74f7cf46-5bf1-4093-931c-ff64378b3814\" (UID: \"74f7cf46-5bf1-4093-931c-ff64378b3814\") "
Sep 16 17:59:52.405169 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.405009    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74f7cf46-5bf1-4093-931c-ff64378b3814-dshm\") pod \"74f7cf46-5bf1-4093-931c-ff64378b3814\" (UID: \"74f7cf46-5bf1-4093-931c-ff64378b3814\") "
Sep 16 17:59:52.405169 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.405049    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74f7cf46-5bf1-4093-931c-ff64378b3814-tls-certs\") pod \"74f7cf46-5bf1-4093-931c-ff64378b3814\" (UID: \"74f7cf46-5bf1-4093-931c-ff64378b3814\") "
Sep 16 17:59:52.405169 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.405074    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74f7cf46-5bf1-4093-931c-ff64378b3814-tmp-dir\") pod \"74f7cf46-5bf1-4093-931c-ff64378b3814\" (UID: \"74f7cf46-5bf1-4093-931c-ff64378b3814\") "
Sep 16 17:59:52.405386 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.405202    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f7cf46-5bf1-4093-931c-ff64378b3814-home" (OuterVolumeSpecName: "home") pod "74f7cf46-5bf1-4093-931c-ff64378b3814" (UID: "74f7cf46-5bf1-4093-931c-ff64378b3814"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:59:52.405386 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.405231    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f7cf46-5bf1-4093-931c-ff64378b3814-model-cache" (OuterVolumeSpecName: "model-cache") pod "74f7cf46-5bf1-4093-931c-ff64378b3814" (UID: "74f7cf46-5bf1-4093-931c-ff64378b3814"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:59:52.405386 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.405325    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74f7cf46-5bf1-4093-931c-ff64378b3814-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:59:52.405386 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.405384    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74f7cf46-5bf1-4093-931c-ff64378b3814-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:59:52.405549 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.405415    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f7cf46-5bf1-4093-931c-ff64378b3814-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "74f7cf46-5bf1-4093-931c-ff64378b3814" (UID: "74f7cf46-5bf1-4093-931c-ff64378b3814"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:59:52.407337 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.407304    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f7cf46-5bf1-4093-931c-ff64378b3814-dshm" (OuterVolumeSpecName: "dshm") pod "74f7cf46-5bf1-4093-931c-ff64378b3814" (UID: "74f7cf46-5bf1-4093-931c-ff64378b3814"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:59:52.407452 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.407361    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f7cf46-5bf1-4093-931c-ff64378b3814-kube-api-access-v7gw4" (OuterVolumeSpecName: "kube-api-access-v7gw4") pod "74f7cf46-5bf1-4093-931c-ff64378b3814" (UID: "74f7cf46-5bf1-4093-931c-ff64378b3814"). InnerVolumeSpecName "kube-api-access-v7gw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:59:52.407452 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.407377    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74f7cf46-5bf1-4093-931c-ff64378b3814-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "74f7cf46-5bf1-4093-931c-ff64378b3814" (UID: "74f7cf46-5bf1-4093-931c-ff64378b3814"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:59:52.506262 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.506229    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74f7cf46-5bf1-4093-931c-ff64378b3814-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:59:52.506262 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.506258    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74f7cf46-5bf1-4093-931c-ff64378b3814-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:59:52.506262 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.506268    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7gw4\" (UniqueName: \"kubernetes.io/projected/74f7cf46-5bf1-4093-931c-ff64378b3814-kube-api-access-v7gw4\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:59:52.506489 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.506276    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74f7cf46-5bf1-4093-931c-ff64378b3814-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 17:59:52.949452 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.949422    2577 generic.go:356] "Generic (PLEG): container finished" podID="74f7cf46-5bf1-4093-931c-ff64378b3814" containerID="84e59760b4b18715f60ca5a37017d117637cfd2b06a88f5a7dff4d37bf2faf48" exitCode=0
Sep 16 17:59:52.949452 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.949448    2577 generic.go:356] "Generic (PLEG): container finished" podID="74f7cf46-5bf1-4093-931c-ff64378b3814" containerID="198102b4bccd946e7fea11c5c2f86220424541cc4ea8c95526819613fd2dce92" exitCode=0
Sep 16 17:59:52.950019 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.949476    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"
Sep 16 17:59:52.950019 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.949486    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6" event={"ID":"74f7cf46-5bf1-4093-931c-ff64378b3814","Type":"ContainerDied","Data":"84e59760b4b18715f60ca5a37017d117637cfd2b06a88f5a7dff4d37bf2faf48"}
Sep 16 17:59:52.950019 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.949544    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6" event={"ID":"74f7cf46-5bf1-4093-931c-ff64378b3814","Type":"ContainerDied","Data":"198102b4bccd946e7fea11c5c2f86220424541cc4ea8c95526819613fd2dce92"}
Sep 16 17:59:52.950019 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.949575    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6" event={"ID":"74f7cf46-5bf1-4093-931c-ff64378b3814","Type":"ContainerDied","Data":"6d265e987d4e14223cbd626b8200b4541395347f4e77ca29b501e24165767140"}
Sep 16 17:59:52.950019 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.949615    2577 scope.go:117] "RemoveContainer" containerID="84e59760b4b18715f60ca5a37017d117637cfd2b06a88f5a7dff4d37bf2faf48"
Sep 16 17:59:52.958897 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.958857    2577 scope.go:117] "RemoveContainer" containerID="198102b4bccd946e7fea11c5c2f86220424541cc4ea8c95526819613fd2dce92"
Sep 16 17:59:52.966908 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.966885    2577 scope.go:117] "RemoveContainer" containerID="84e59760b4b18715f60ca5a37017d117637cfd2b06a88f5a7dff4d37bf2faf48"
Sep 16 17:59:52.967154 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:59:52.967131    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84e59760b4b18715f60ca5a37017d117637cfd2b06a88f5a7dff4d37bf2faf48\": container with ID starting with 84e59760b4b18715f60ca5a37017d117637cfd2b06a88f5a7dff4d37bf2faf48 not found: ID does not exist" containerID="84e59760b4b18715f60ca5a37017d117637cfd2b06a88f5a7dff4d37bf2faf48"
Sep 16 17:59:52.967237 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.967161    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e59760b4b18715f60ca5a37017d117637cfd2b06a88f5a7dff4d37bf2faf48"} err="failed to get container status \"84e59760b4b18715f60ca5a37017d117637cfd2b06a88f5a7dff4d37bf2faf48\": rpc error: code = NotFound desc = could not find container \"84e59760b4b18715f60ca5a37017d117637cfd2b06a88f5a7dff4d37bf2faf48\": container with ID starting with 84e59760b4b18715f60ca5a37017d117637cfd2b06a88f5a7dff4d37bf2faf48 not found: ID does not exist"
Sep 16 17:59:52.967237 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.967178    2577 scope.go:117] "RemoveContainer" containerID="198102b4bccd946e7fea11c5c2f86220424541cc4ea8c95526819613fd2dce92"
Sep 16 17:59:52.967463 ip-10-0-136-122 kubenswrapper[2577]: E0916 17:59:52.967447    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"198102b4bccd946e7fea11c5c2f86220424541cc4ea8c95526819613fd2dce92\": container with ID starting with 198102b4bccd946e7fea11c5c2f86220424541cc4ea8c95526819613fd2dce92 not found: ID does not exist" containerID="198102b4bccd946e7fea11c5c2f86220424541cc4ea8c95526819613fd2dce92"
Sep 16 17:59:52.967512 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.967469    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"198102b4bccd946e7fea11c5c2f86220424541cc4ea8c95526819613fd2dce92"} err="failed to get container status \"198102b4bccd946e7fea11c5c2f86220424541cc4ea8c95526819613fd2dce92\": rpc error: code = NotFound desc = could not find container \"198102b4bccd946e7fea11c5c2f86220424541cc4ea8c95526819613fd2dce92\": container with ID starting with 198102b4bccd946e7fea11c5c2f86220424541cc4ea8c95526819613fd2dce92 not found: ID does not exist"
Sep 16 17:59:52.967512 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.967481    2577 scope.go:117] "RemoveContainer" containerID="84e59760b4b18715f60ca5a37017d117637cfd2b06a88f5a7dff4d37bf2faf48"
Sep 16 17:59:52.967729 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.967707    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e59760b4b18715f60ca5a37017d117637cfd2b06a88f5a7dff4d37bf2faf48"} err="failed to get container status \"84e59760b4b18715f60ca5a37017d117637cfd2b06a88f5a7dff4d37bf2faf48\": rpc error: code = NotFound desc = could not find container \"84e59760b4b18715f60ca5a37017d117637cfd2b06a88f5a7dff4d37bf2faf48\": container with ID starting with 84e59760b4b18715f60ca5a37017d117637cfd2b06a88f5a7dff4d37bf2faf48 not found: ID does not exist"
Sep 16 17:59:52.967779 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.967730    2577 scope.go:117] "RemoveContainer" containerID="198102b4bccd946e7fea11c5c2f86220424541cc4ea8c95526819613fd2dce92"
Sep 16 17:59:52.967983 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.967966    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"198102b4bccd946e7fea11c5c2f86220424541cc4ea8c95526819613fd2dce92"} err="failed to get container status \"198102b4bccd946e7fea11c5c2f86220424541cc4ea8c95526819613fd2dce92\": rpc error: code = NotFound desc = could not find container \"198102b4bccd946e7fea11c5c2f86220424541cc4ea8c95526819613fd2dce92\": container with ID starting with 198102b4bccd946e7fea11c5c2f86220424541cc4ea8c95526819613fd2dce92 not found: ID does not exist"
Sep 16 17:59:52.988908 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.988848    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"]
Sep 16 17:59:52.993523 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:52.993491    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-rcxw6"]
Sep 16 17:59:53.080785 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:53.080755    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74f7cf46-5bf1-4093-931c-ff64378b3814" path="/var/lib/kubelet/pods/74f7cf46-5bf1-4093-931c-ff64378b3814/volumes"
Sep 16 17:59:56.082450 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:56.082412    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:59:56.269810 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:56.269746    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 17:59:56.375179 ip-10-0-136-122 kubenswrapper[2577]: I0916 17:59:56.375095    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:00:02.478676 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:02.478634    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:00:02.577392 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:02.577351    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:00:08.847903 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:08.847853    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94"]
Sep 16 18:00:08.860857 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:08.860820    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:00:23.891551 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:23.891513    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94" podUID="125127f8-862a-4225-b042-16f0d13c377c" containerName="main" containerID="cri-o://55ee61bf5d923b3fd98ad5119f8315cbfbead04941dd95c44a6809c02fca24c2" gracePeriod=45
Sep 16 18:00:24.113900 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:24.113850    2577 generic.go:356] "Generic (PLEG): container finished" podID="125127f8-862a-4225-b042-16f0d13c377c" containerID="55ee61bf5d923b3fd98ad5119f8315cbfbead04941dd95c44a6809c02fca24c2" exitCode=0
Sep 16 18:00:24.114093 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:24.113897    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94" event={"ID":"125127f8-862a-4225-b042-16f0d13c377c","Type":"ContainerDied","Data":"55ee61bf5d923b3fd98ad5119f8315cbfbead04941dd95c44a6809c02fca24c2"}
Sep 16 18:00:24.146184 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:24.146131    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94"
Sep 16 18:00:24.276475 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:24.276439    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/125127f8-862a-4225-b042-16f0d13c377c-tls-certs\") pod \"125127f8-862a-4225-b042-16f0d13c377c\" (UID: \"125127f8-862a-4225-b042-16f0d13c377c\") "
Sep 16 18:00:24.276475 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:24.276477    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7xkb\" (UniqueName: \"kubernetes.io/projected/125127f8-862a-4225-b042-16f0d13c377c-kube-api-access-q7xkb\") pod \"125127f8-862a-4225-b042-16f0d13c377c\" (UID: \"125127f8-862a-4225-b042-16f0d13c377c\") "
Sep 16 18:00:24.276735 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:24.276511    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/125127f8-862a-4225-b042-16f0d13c377c-model-cache\") pod \"125127f8-862a-4225-b042-16f0d13c377c\" (UID: \"125127f8-862a-4225-b042-16f0d13c377c\") "
Sep 16 18:00:24.276735 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:24.276585    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/125127f8-862a-4225-b042-16f0d13c377c-dshm\") pod \"125127f8-862a-4225-b042-16f0d13c377c\" (UID: \"125127f8-862a-4225-b042-16f0d13c377c\") "
Sep 16 18:00:24.276735 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:24.276606    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/125127f8-862a-4225-b042-16f0d13c377c-home\") pod \"125127f8-862a-4225-b042-16f0d13c377c\" (UID: \"125127f8-862a-4225-b042-16f0d13c377c\") "
Sep 16 18:00:24.276735 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:24.276627    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/125127f8-862a-4225-b042-16f0d13c377c-tmp-dir\") pod \"125127f8-862a-4225-b042-16f0d13c377c\" (UID: \"125127f8-862a-4225-b042-16f0d13c377c\") "
Sep 16 18:00:24.276978 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:24.276837    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/125127f8-862a-4225-b042-16f0d13c377c-model-cache" (OuterVolumeSpecName: "model-cache") pod "125127f8-862a-4225-b042-16f0d13c377c" (UID: "125127f8-862a-4225-b042-16f0d13c377c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:00:24.276978 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:24.276885    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/125127f8-862a-4225-b042-16f0d13c377c-home" (OuterVolumeSpecName: "home") pod "125127f8-862a-4225-b042-16f0d13c377c" (UID: "125127f8-862a-4225-b042-16f0d13c377c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:00:24.277048 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:24.276985    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/125127f8-862a-4225-b042-16f0d13c377c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "125127f8-862a-4225-b042-16f0d13c377c" (UID: "125127f8-862a-4225-b042-16f0d13c377c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:00:24.278853 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:24.278828    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/125127f8-862a-4225-b042-16f0d13c377c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "125127f8-862a-4225-b042-16f0d13c377c" (UID: "125127f8-862a-4225-b042-16f0d13c377c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:00:24.278958 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:24.278895    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/125127f8-862a-4225-b042-16f0d13c377c-kube-api-access-q7xkb" (OuterVolumeSpecName: "kube-api-access-q7xkb") pod "125127f8-862a-4225-b042-16f0d13c377c" (UID: "125127f8-862a-4225-b042-16f0d13c377c"). InnerVolumeSpecName "kube-api-access-q7xkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:00:24.278958 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:24.278921    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/125127f8-862a-4225-b042-16f0d13c377c-dshm" (OuterVolumeSpecName: "dshm") pod "125127f8-862a-4225-b042-16f0d13c377c" (UID: "125127f8-862a-4225-b042-16f0d13c377c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:00:24.377802 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:24.377770    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/125127f8-862a-4225-b042-16f0d13c377c-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:00:24.377802 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:24.377799    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/125127f8-862a-4225-b042-16f0d13c377c-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:00:24.377802 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:24.377807    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/125127f8-862a-4225-b042-16f0d13c377c-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:00:24.378043 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:24.377817    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/125127f8-862a-4225-b042-16f0d13c377c-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:00:24.378043 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:24.377826    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7xkb\" (UniqueName: \"kubernetes.io/projected/125127f8-862a-4225-b042-16f0d13c377c-kube-api-access-q7xkb\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:00:24.378043 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:24.377834    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/125127f8-862a-4225-b042-16f0d13c377c-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:00:25.120332 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:25.120295    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94" event={"ID":"125127f8-862a-4225-b042-16f0d13c377c","Type":"ContainerDied","Data":"f63fdd7b2cdb91206d08a1242912109b0cf1402f832140e305bd872ff3683ee6"}
Sep 16 18:00:25.120773 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:25.120335    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94"
Sep 16 18:00:25.120773 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:25.120364    2577 scope.go:117] "RemoveContainer" containerID="55ee61bf5d923b3fd98ad5119f8315cbfbead04941dd95c44a6809c02fca24c2"
Sep 16 18:00:25.138700 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:25.138676    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94"]
Sep 16 18:00:25.143116 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:25.143092    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-dcz94"]
Sep 16 18:00:27.080716 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:27.080682    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="125127f8-862a-4225-b042-16f0d13c377c" path="/var/lib/kubelet/pods/125127f8-862a-4225-b042-16f0d13c377c/volumes"
Sep 16 18:00:30.671983 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:30.671945    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:00:30.836246 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:30.836208    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:00:34.920060 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:34.919979    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:00:36.153003 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.152968    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284"]
Sep 16 18:00:36.153393 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.153330    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="125127f8-862a-4225-b042-16f0d13c377c" containerName="main"
Sep 16 18:00:36.153393 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.153342    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="125127f8-862a-4225-b042-16f0d13c377c" containerName="main"
Sep 16 18:00:36.153393 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.153353    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74f7cf46-5bf1-4093-931c-ff64378b3814" containerName="llm-d-routing-sidecar"
Sep 16 18:00:36.153393 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.153360    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="74f7cf46-5bf1-4093-931c-ff64378b3814" containerName="llm-d-routing-sidecar"
Sep 16 18:00:36.153393 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.153375    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ce1eb56-2a7e-4564-8bf6-1156b9b75512" containerName="main"
Sep 16 18:00:36.153393 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.153381    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ce1eb56-2a7e-4564-8bf6-1156b9b75512" containerName="main"
Sep 16 18:00:36.153393 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.153388    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="107a52f1-5dc6-4b42-ad91-32c350583beb" containerName="main"
Sep 16 18:00:36.153393 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.153394    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="107a52f1-5dc6-4b42-ad91-32c350583beb" containerName="main"
Sep 16 18:00:36.153649 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.153407    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74f7cf46-5bf1-4093-931c-ff64378b3814" containerName="main"
Sep 16 18:00:36.153649 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.153412    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="74f7cf46-5bf1-4093-931c-ff64378b3814" containerName="main"
Sep 16 18:00:36.153649 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.153476    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="107a52f1-5dc6-4b42-ad91-32c350583beb" containerName="main"
Sep 16 18:00:36.153649 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.153484    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ce1eb56-2a7e-4564-8bf6-1156b9b75512" containerName="main"
Sep 16 18:00:36.153649 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.153489    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="74f7cf46-5bf1-4093-931c-ff64378b3814" containerName="main"
Sep 16 18:00:36.153649 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.153495    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="74f7cf46-5bf1-4093-931c-ff64378b3814" containerName="llm-d-routing-sidecar"
Sep 16 18:00:36.153649 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.153502    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="125127f8-862a-4225-b042-16f0d13c377c" containerName="main"
Sep 16 18:00:36.156495 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.156478    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284"
Sep 16 18:00:36.159962 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.159939    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"kube-root-ca.crt\""
Sep 16 18:00:36.160082 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.159950    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"openshift-service-ca.crt\""
Sep 16 18:00:36.160510 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.160493    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"default-dockercfg-7l5jz\""
Sep 16 18:00:36.160629 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.160586    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v1-kserve-self-signed-certs\""
Sep 16 18:00:36.174192 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.174151    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284"]
Sep 16 18:00:36.195069 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.195029    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:00:36.280936 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.280894    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/582e8843-4a36-4b0d-96ca-c065aec4d6c7-dshm\") pod \"decom-v1-kserve-6cb797fb75-8z284\" (UID: \"582e8843-4a36-4b0d-96ca-c065aec4d6c7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284"
Sep 16 18:00:36.281117 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.280958    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csth6\" (UniqueName: \"kubernetes.io/projected/582e8843-4a36-4b0d-96ca-c065aec4d6c7-kube-api-access-csth6\") pod \"decom-v1-kserve-6cb797fb75-8z284\" (UID: \"582e8843-4a36-4b0d-96ca-c065aec4d6c7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284"
Sep 16 18:00:36.281117 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.281017    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/582e8843-4a36-4b0d-96ca-c065aec4d6c7-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-8z284\" (UID: \"582e8843-4a36-4b0d-96ca-c065aec4d6c7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284"
Sep 16 18:00:36.281117 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.281054    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/582e8843-4a36-4b0d-96ca-c065aec4d6c7-model-cache\") pod \"decom-v1-kserve-6cb797fb75-8z284\" (UID: \"582e8843-4a36-4b0d-96ca-c065aec4d6c7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284"
Sep 16 18:00:36.281117 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.281097    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/582e8843-4a36-4b0d-96ca-c065aec4d6c7-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-8z284\" (UID: \"582e8843-4a36-4b0d-96ca-c065aec4d6c7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284"
Sep 16 18:00:36.281264 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.281169    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/582e8843-4a36-4b0d-96ca-c065aec4d6c7-home\") pod \"decom-v1-kserve-6cb797fb75-8z284\" (UID: \"582e8843-4a36-4b0d-96ca-c065aec4d6c7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284"
Sep 16 18:00:36.382448 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.382405    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/582e8843-4a36-4b0d-96ca-c065aec4d6c7-home\") pod \"decom-v1-kserve-6cb797fb75-8z284\" (UID: \"582e8843-4a36-4b0d-96ca-c065aec4d6c7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284"
Sep 16 18:00:36.382448 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.382447    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/582e8843-4a36-4b0d-96ca-c065aec4d6c7-dshm\") pod \"decom-v1-kserve-6cb797fb75-8z284\" (UID: \"582e8843-4a36-4b0d-96ca-c065aec4d6c7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284"
Sep 16 18:00:36.382711 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.382476    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csth6\" (UniqueName: \"kubernetes.io/projected/582e8843-4a36-4b0d-96ca-c065aec4d6c7-kube-api-access-csth6\") pod \"decom-v1-kserve-6cb797fb75-8z284\" (UID: \"582e8843-4a36-4b0d-96ca-c065aec4d6c7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284"
Sep 16 18:00:36.382711 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.382597    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/582e8843-4a36-4b0d-96ca-c065aec4d6c7-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-8z284\" (UID: \"582e8843-4a36-4b0d-96ca-c065aec4d6c7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284"
Sep 16 18:00:36.382711 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.382658    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/582e8843-4a36-4b0d-96ca-c065aec4d6c7-model-cache\") pod \"decom-v1-kserve-6cb797fb75-8z284\" (UID: \"582e8843-4a36-4b0d-96ca-c065aec4d6c7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284"
Sep 16 18:00:36.382887 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.382725    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/582e8843-4a36-4b0d-96ca-c065aec4d6c7-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-8z284\" (UID: \"582e8843-4a36-4b0d-96ca-c065aec4d6c7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284"
Sep 16 18:00:36.382887 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.382852    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/582e8843-4a36-4b0d-96ca-c065aec4d6c7-home\") pod \"decom-v1-kserve-6cb797fb75-8z284\" (UID: \"582e8843-4a36-4b0d-96ca-c065aec4d6c7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284"
Sep 16 18:00:36.383031 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.383011    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/582e8843-4a36-4b0d-96ca-c065aec4d6c7-model-cache\") pod \"decom-v1-kserve-6cb797fb75-8z284\" (UID: \"582e8843-4a36-4b0d-96ca-c065aec4d6c7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284"
Sep 16 18:00:36.383125 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.383106    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/582e8843-4a36-4b0d-96ca-c065aec4d6c7-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-8z284\" (UID: \"582e8843-4a36-4b0d-96ca-c065aec4d6c7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284"
Sep 16 18:00:36.384696 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.384679    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/582e8843-4a36-4b0d-96ca-c065aec4d6c7-dshm\") pod \"decom-v1-kserve-6cb797fb75-8z284\" (UID: \"582e8843-4a36-4b0d-96ca-c065aec4d6c7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284"
Sep 16 18:00:36.385142 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.385121    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/582e8843-4a36-4b0d-96ca-c065aec4d6c7-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-8z284\" (UID: \"582e8843-4a36-4b0d-96ca-c065aec4d6c7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284"
Sep 16 18:00:36.392176 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.392150    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csth6\" (UniqueName: \"kubernetes.io/projected/582e8843-4a36-4b0d-96ca-c065aec4d6c7-kube-api-access-csth6\") pod \"decom-v1-kserve-6cb797fb75-8z284\" (UID: \"582e8843-4a36-4b0d-96ca-c065aec4d6c7\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284"
Sep 16 18:00:36.473688 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.473581    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284"
Sep 16 18:00:36.752716 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:36.752687    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284"]
Sep 16 18:00:36.754831 ip-10-0-136-122 kubenswrapper[2577]: W0916 18:00:36.754782    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod582e8843_4a36_4b0d_96ca_c065aec4d6c7.slice/crio-8e879e6dce29bd65506b847ecbd5a0a36e7865fe3cb925a98768816d3d9a2290 WatchSource:0}: Error finding container 8e879e6dce29bd65506b847ecbd5a0a36e7865fe3cb925a98768816d3d9a2290: Status 404 returned error can't find the container with id 8e879e6dce29bd65506b847ecbd5a0a36e7865fe3cb925a98768816d3d9a2290
Sep 16 18:00:37.184479 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:37.184441    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284" event={"ID":"582e8843-4a36-4b0d-96ca-c065aec4d6c7","Type":"ContainerStarted","Data":"3dcbee409330cd37b7506954e7bdab44bf1301e205ac6a562b6dd3d074966f9f"}
Sep 16 18:00:37.184849 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:37.184507    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284" event={"ID":"582e8843-4a36-4b0d-96ca-c065aec4d6c7","Type":"ContainerStarted","Data":"8e879e6dce29bd65506b847ecbd5a0a36e7865fe3cb925a98768816d3d9a2290"}
Sep 16 18:00:37.206416 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:37.206371    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284" podStartSLOduration=1.20635531 podStartE2EDuration="1.20635531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:00:37.204726951 +0000 UTC m=+2158.567405737" watchObservedRunningTime="2026-09-16 18:00:37.20635531 +0000 UTC m=+2158.569034095"
Sep 16 18:00:37.474652 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:37.474570    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284"
Sep 16 18:00:38.730638 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:38.730601    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:00:38.836653 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:38.836616    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:00:44.891148 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:44.891096    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:00:45.611466 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:45.611429    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:00:46.474576 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:46.474538    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284"
Sep 16 18:00:46.487058 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:46.487028    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284"
Sep 16 18:00:47.250524 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:47.250496    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284"
Sep 16 18:00:51.445305 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:51.445271    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:00:51.545344 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:51.545304    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:00:59.645419 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.645377    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"]
Sep 16 18:00:59.649273 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.649249    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"
Sep 16 18:00:59.652055 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.652036    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 16 18:00:59.653151 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.653130    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 16 18:00:59.653263 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.653145    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 16 18:00:59.653331 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.653195    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-8xfng\""
Sep 16 18:00:59.653397 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.653207    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 16 18:00:59.653397 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.653208    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:00:59.664250 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.664228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c5c997d-c7d0-4366-84c1-2065e82bedcd-model-cache\") pod \"tokenizer-clean-path-test-kserve-d975798d6-fvp8v\" (UID: \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"
Sep 16 18:00:59.664359 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.664261    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c5c997d-c7d0-4366-84c1-2065e82bedcd-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-d975798d6-fvp8v\" (UID: \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"
Sep 16 18:00:59.664359 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.664292    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c5c997d-c7d0-4366-84c1-2065e82bedcd-home\") pod \"tokenizer-clean-path-test-kserve-d975798d6-fvp8v\" (UID: \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"
Sep 16 18:00:59.664359 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.664352    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c5c997d-c7d0-4366-84c1-2065e82bedcd-dshm\") pod \"tokenizer-clean-path-test-kserve-d975798d6-fvp8v\" (UID: \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"
Sep 16 18:00:59.664523 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.664393    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c5c997d-c7d0-4366-84c1-2065e82bedcd-tls-certs\") pod \"tokenizer-clean-path-test-kserve-d975798d6-fvp8v\" (UID: \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"
Sep 16 18:00:59.664523 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.664462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c5c997d-c7d0-4366-84c1-2065e82bedcd-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-d975798d6-fvp8v\" (UID: \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"
Sep 16 18:00:59.664523 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.664506    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c5c997d-c7d0-4366-84c1-2065e82bedcd-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-d975798d6-fvp8v\" (UID: \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"
Sep 16 18:00:59.664646 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.664535    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzj95\" (UniqueName: \"kubernetes.io/projected/3c5c997d-c7d0-4366-84c1-2065e82bedcd-kube-api-access-kzj95\") pod \"tokenizer-clean-path-test-kserve-d975798d6-fvp8v\" (UID: \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"
Sep 16 18:00:59.669365 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.669341    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"]
Sep 16 18:00:59.765104 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.765053    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c5c997d-c7d0-4366-84c1-2065e82bedcd-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-d975798d6-fvp8v\" (UID: \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"
Sep 16 18:00:59.765104 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.765107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c5c997d-c7d0-4366-84c1-2065e82bedcd-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-d975798d6-fvp8v\" (UID: \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"
Sep 16 18:00:59.765325 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.765228    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzj95\" (UniqueName: \"kubernetes.io/projected/3c5c997d-c7d0-4366-84c1-2065e82bedcd-kube-api-access-kzj95\") pod \"tokenizer-clean-path-test-kserve-d975798d6-fvp8v\" (UID: \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"
Sep 16 18:00:59.765388 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.765368    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c5c997d-c7d0-4366-84c1-2065e82bedcd-model-cache\") pod \"tokenizer-clean-path-test-kserve-d975798d6-fvp8v\" (UID: \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"
Sep 16 18:00:59.765425 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.765405    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c5c997d-c7d0-4366-84c1-2065e82bedcd-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-d975798d6-fvp8v\" (UID: \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"
Sep 16 18:00:59.765463 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.765420    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c5c997d-c7d0-4366-84c1-2065e82bedcd-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-d975798d6-fvp8v\" (UID: \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"
Sep 16 18:00:59.765496 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.765479    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c5c997d-c7d0-4366-84c1-2065e82bedcd-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-d975798d6-fvp8v\" (UID: \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"
Sep 16 18:00:59.765750 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.765728    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c5c997d-c7d0-4366-84c1-2065e82bedcd-home\") pod \"tokenizer-clean-path-test-kserve-d975798d6-fvp8v\" (UID: \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"
Sep 16 18:00:59.765979 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.765957    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c5c997d-c7d0-4366-84c1-2065e82bedcd-dshm\") pod \"tokenizer-clean-path-test-kserve-d975798d6-fvp8v\" (UID: \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"
Sep 16 18:00:59.766094 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.766013    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c5c997d-c7d0-4366-84c1-2065e82bedcd-tls-certs\") pod \"tokenizer-clean-path-test-kserve-d975798d6-fvp8v\" (UID: \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"
Sep 16 18:00:59.769898 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.766181    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c5c997d-c7d0-4366-84c1-2065e82bedcd-home\") pod \"tokenizer-clean-path-test-kserve-d975798d6-fvp8v\" (UID: \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"
Sep 16 18:00:59.769898 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.766233    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c5c997d-c7d0-4366-84c1-2065e82bedcd-model-cache\") pod \"tokenizer-clean-path-test-kserve-d975798d6-fvp8v\" (UID: \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"
Sep 16 18:00:59.769898 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.766623    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c5c997d-c7d0-4366-84c1-2065e82bedcd-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-d975798d6-fvp8v\" (UID: \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"
Sep 16 18:00:59.769898 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.768759    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c5c997d-c7d0-4366-84c1-2065e82bedcd-dshm\") pod \"tokenizer-clean-path-test-kserve-d975798d6-fvp8v\" (UID: \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"
Sep 16 18:00:59.769898 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.769180    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c5c997d-c7d0-4366-84c1-2065e82bedcd-tls-certs\") pod \"tokenizer-clean-path-test-kserve-d975798d6-fvp8v\" (UID: \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"
Sep 16 18:00:59.775099 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.775073    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzj95\" (UniqueName: \"kubernetes.io/projected/3c5c997d-c7d0-4366-84c1-2065e82bedcd-kube-api-access-kzj95\") pod \"tokenizer-clean-path-test-kserve-d975798d6-fvp8v\" (UID: \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"
Sep 16 18:00:59.962179 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.962089    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"
Sep 16 18:00:59.978693 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.978652    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b"]
Sep 16 18:00:59.983399 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:00:59.983374    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b"
Sep 16 18:01:00.001317 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:00.001285    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b"]
Sep 16 18:01:00.040442 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:00.040394    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:01:00.068378 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:00.068330    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be8d290e-7584-426c-91fd-9e1b1e1bdad6-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b\" (UID: \"be8d290e-7584-426c-91fd-9e1b1e1bdad6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b"
Sep 16 18:01:00.068537 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:00.068393    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be8d290e-7584-426c-91fd-9e1b1e1bdad6-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b\" (UID: \"be8d290e-7584-426c-91fd-9e1b1e1bdad6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b"
Sep 16 18:01:00.068537 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:00.068424    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be8d290e-7584-426c-91fd-9e1b1e1bdad6-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b\" (UID: \"be8d290e-7584-426c-91fd-9e1b1e1bdad6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b"
Sep 16 18:01:00.068537 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:00.068482    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be8d290e-7584-426c-91fd-9e1b1e1bdad6-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b\" (UID: \"be8d290e-7584-426c-91fd-9e1b1e1bdad6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b"
Sep 16 18:01:00.140855 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:00.140814    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:01:00.169021 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:00.168989    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be8d290e-7584-426c-91fd-9e1b1e1bdad6-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b\" (UID: \"be8d290e-7584-426c-91fd-9e1b1e1bdad6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b"
Sep 16 18:01:00.169169 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:00.169041    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be8d290e-7584-426c-91fd-9e1b1e1bdad6-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b\" (UID: \"be8d290e-7584-426c-91fd-9e1b1e1bdad6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b"
Sep 16 18:01:00.169169 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:00.169077    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be8d290e-7584-426c-91fd-9e1b1e1bdad6-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b\" (UID: \"be8d290e-7584-426c-91fd-9e1b1e1bdad6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b"
Sep 16 18:01:00.169169 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:00.169098    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be8d290e-7584-426c-91fd-9e1b1e1bdad6-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b\" (UID: \"be8d290e-7584-426c-91fd-9e1b1e1bdad6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b"
Sep 16 18:01:00.169558 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:00.169538    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be8d290e-7584-426c-91fd-9e1b1e1bdad6-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b\" (UID: \"be8d290e-7584-426c-91fd-9e1b1e1bdad6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b"
Sep 16 18:01:00.169633 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:00.169570    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be8d290e-7584-426c-91fd-9e1b1e1bdad6-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b\" (UID: \"be8d290e-7584-426c-91fd-9e1b1e1bdad6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b"
Sep 16 18:01:00.169859 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:00.169840    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be8d290e-7584-426c-91fd-9e1b1e1bdad6-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b\" (UID: \"be8d290e-7584-426c-91fd-9e1b1e1bdad6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b"
Sep 16 18:01:00.171520 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:00.171499    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be8d290e-7584-426c-91fd-9e1b1e1bdad6-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b\" (UID: \"be8d290e-7584-426c-91fd-9e1b1e1bdad6\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b"
Sep 16 18:01:00.265972 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:00.265939    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"]
Sep 16 18:01:00.267757 ip-10-0-136-122 kubenswrapper[2577]: W0916 18:01:00.267730    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c5c997d_c7d0_4366_84c1_2065e82bedcd.slice/crio-3eb65d337cf881230104ea28bf5fd3ad32e40bc4fc36825e78b76edd5fdd770f WatchSource:0}: Error finding container 3eb65d337cf881230104ea28bf5fd3ad32e40bc4fc36825e78b76edd5fdd770f: Status 404 returned error can't find the container with id 3eb65d337cf881230104ea28bf5fd3ad32e40bc4fc36825e78b76edd5fdd770f
Sep 16 18:01:00.302040 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:00.302010    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b"
Sep 16 18:01:00.309951 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:00.309261    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v" event={"ID":"3c5c997d-c7d0-4366-84c1-2065e82bedcd","Type":"ContainerStarted","Data":"3eb65d337cf881230104ea28bf5fd3ad32e40bc4fc36825e78b76edd5fdd770f"}
Sep 16 18:01:00.589357 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:00.589326    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b"]
Sep 16 18:01:00.591665 ip-10-0-136-122 kubenswrapper[2577]: W0916 18:01:00.591627    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe8d290e_7584_426c_91fd_9e1b1e1bdad6.slice/crio-04bfd18a14a461d5d46020443c4912a56dc3ac02b95bd8171355ab3254f39a83 WatchSource:0}: Error finding container 04bfd18a14a461d5d46020443c4912a56dc3ac02b95bd8171355ab3254f39a83: Status 404 returned error can't find the container with id 04bfd18a14a461d5d46020443c4912a56dc3ac02b95bd8171355ab3254f39a83
Sep 16 18:01:01.318281 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:01.318234    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b" event={"ID":"be8d290e-7584-426c-91fd-9e1b1e1bdad6","Type":"ContainerStarted","Data":"c370384188f34926b995952c34c49ecfdd4399fac0b2444d3ed7d947be81335d"}
Sep 16 18:01:01.318710 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:01.318302    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b" event={"ID":"be8d290e-7584-426c-91fd-9e1b1e1bdad6","Type":"ContainerStarted","Data":"04bfd18a14a461d5d46020443c4912a56dc3ac02b95bd8171355ab3254f39a83"}
Sep 16 18:01:01.319960 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:01.319931    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v" event={"ID":"3c5c997d-c7d0-4366-84c1-2065e82bedcd","Type":"ContainerStarted","Data":"9ac3ca113ee3906cc8ff7c85b66156d74a76b984e9c5c9f5211e875086ecbf78"}
Sep 16 18:01:03.337558 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:03.337524    2577 generic.go:356] "Generic (PLEG): container finished" podID="3c5c997d-c7d0-4366-84c1-2065e82bedcd" containerID="9ac3ca113ee3906cc8ff7c85b66156d74a76b984e9c5c9f5211e875086ecbf78" exitCode=0
Sep 16 18:01:03.337985 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:03.337599    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v" event={"ID":"3c5c997d-c7d0-4366-84c1-2065e82bedcd","Type":"ContainerDied","Data":"9ac3ca113ee3906cc8ff7c85b66156d74a76b984e9c5c9f5211e875086ecbf78"}
Sep 16 18:01:03.339406 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:03.339382    2577 generic.go:356] "Generic (PLEG): container finished" podID="be8d290e-7584-426c-91fd-9e1b1e1bdad6" containerID="c370384188f34926b995952c34c49ecfdd4399fac0b2444d3ed7d947be81335d" exitCode=0
Sep 16 18:01:03.339500 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:03.339426    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b" event={"ID":"be8d290e-7584-426c-91fd-9e1b1e1bdad6","Type":"ContainerDied","Data":"c370384188f34926b995952c34c49ecfdd4399fac0b2444d3ed7d947be81335d"}
Sep 16 18:01:03.340446 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:03.340426    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:01:04.348701 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:04.348661    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v" event={"ID":"3c5c997d-c7d0-4366-84c1-2065e82bedcd","Type":"ContainerStarted","Data":"b4b48c0d96e73e8dc17ab099613d533d679b62038301ceeee624fa4aee7d8e0a"}
Sep 16 18:01:04.373376 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:04.373274    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v" podStartSLOduration=2.857837442 podStartE2EDuration="5.37325714s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.515419698s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:01:04.370958395 +0000 UTC m=+2185.733637181" watchObservedRunningTime="2026-09-16 18:01:04.37325714 +0000 UTC m=+2185.735935928"
Sep 16 18:01:04.962668 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:04.962626    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"
Sep 16 18:01:07.159389 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:07.159333    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:01:09.962978 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:09.962938    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"
Sep 16 18:01:09.982540 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:09.982506    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"
Sep 16 18:01:10.412799 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:10.412756    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"
Sep 16 18:01:12.586025 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:12.585985    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:01:18.670215 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:18.670160    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:01:18.772913 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:18.772828    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:01:31.444644 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:31.444604    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:01:31.538025 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:31.537984    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:01:51.680908 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:51.680873    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b" event={"ID":"be8d290e-7584-426c-91fd-9e1b1e1bdad6","Type":"ContainerStarted","Data":"7633a0318eeab157082c6f062cf635302ddd052ccbe4a42b61eee68f7561e693"}
Sep 16 18:01:51.681556 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:51.681089    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b"
Sep 16 18:01:53.118590 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:53.118553    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:01:58.252658 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:01:58.252603    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:02:12.238199 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:12.238134    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:02:12.341408 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:12.341373    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:02:12.418214 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:12.418155    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b" podStartSLOduration=23.2844784 podStartE2EDuration="1m13.418137718s" totalImagesPullingTime="47.5714691s" totalInitContainerRuntime="2.562190218s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:00:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:01:51.724014037 +0000 UTC m=+2233.086692853" watchObservedRunningTime="2026-09-16 18:02:12.418137718 +0000 UTC m=+2253.780816503"
Sep 16 18:02:12.418947 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:12.418922    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284"]
Sep 16 18:02:12.422450 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:12.422420    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:02:12.502196 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:12.502110    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:02:17.234796 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:17.234758    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:02:17.327799 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:17.327765    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:02:22.698544 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:22.698500    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b"
Sep 16 18:02:27.438692 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:27.438631    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284" podUID="582e8843-4a36-4b0d-96ca-c065aec4d6c7" containerName="main" containerID="cri-o://3dcbee409330cd37b7506954e7bdab44bf1301e205ac6a562b6dd3d074966f9f" gracePeriod=45
Sep 16 18:02:27.694501 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:27.694422    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284"
Sep 16 18:02:27.764782 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:27.764746    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/582e8843-4a36-4b0d-96ca-c065aec4d6c7-dshm\") pod \"582e8843-4a36-4b0d-96ca-c065aec4d6c7\" (UID: \"582e8843-4a36-4b0d-96ca-c065aec4d6c7\") "
Sep 16 18:02:27.764999 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:27.764796    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csth6\" (UniqueName: \"kubernetes.io/projected/582e8843-4a36-4b0d-96ca-c065aec4d6c7-kube-api-access-csth6\") pod \"582e8843-4a36-4b0d-96ca-c065aec4d6c7\" (UID: \"582e8843-4a36-4b0d-96ca-c065aec4d6c7\") "
Sep 16 18:02:27.764999 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:27.764821    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/582e8843-4a36-4b0d-96ca-c065aec4d6c7-home\") pod \"582e8843-4a36-4b0d-96ca-c065aec4d6c7\" (UID: \"582e8843-4a36-4b0d-96ca-c065aec4d6c7\") "
Sep 16 18:02:27.764999 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:27.764889    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/582e8843-4a36-4b0d-96ca-c065aec4d6c7-tmp-dir\") pod \"582e8843-4a36-4b0d-96ca-c065aec4d6c7\" (UID: \"582e8843-4a36-4b0d-96ca-c065aec4d6c7\") "
Sep 16 18:02:27.764999 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:27.764956    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/582e8843-4a36-4b0d-96ca-c065aec4d6c7-tls-certs\") pod \"582e8843-4a36-4b0d-96ca-c065aec4d6c7\" (UID: \"582e8843-4a36-4b0d-96ca-c065aec4d6c7\") "
Sep 16 18:02:27.764999 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:27.764996    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/582e8843-4a36-4b0d-96ca-c065aec4d6c7-model-cache\") pod \"582e8843-4a36-4b0d-96ca-c065aec4d6c7\" (UID: \"582e8843-4a36-4b0d-96ca-c065aec4d6c7\") "
Sep 16 18:02:27.765276 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:27.765132    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582e8843-4a36-4b0d-96ca-c065aec4d6c7-home" (OuterVolumeSpecName: "home") pod "582e8843-4a36-4b0d-96ca-c065aec4d6c7" (UID: "582e8843-4a36-4b0d-96ca-c065aec4d6c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:02:27.765276 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:27.765153    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582e8843-4a36-4b0d-96ca-c065aec4d6c7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "582e8843-4a36-4b0d-96ca-c065aec4d6c7" (UID: "582e8843-4a36-4b0d-96ca-c065aec4d6c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:02:27.765375 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:27.765317    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/582e8843-4a36-4b0d-96ca-c065aec4d6c7-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:02:27.765375 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:27.765331    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/582e8843-4a36-4b0d-96ca-c065aec4d6c7-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:02:27.765375 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:27.765333    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582e8843-4a36-4b0d-96ca-c065aec4d6c7-model-cache" (OuterVolumeSpecName: "model-cache") pod "582e8843-4a36-4b0d-96ca-c065aec4d6c7" (UID: "582e8843-4a36-4b0d-96ca-c065aec4d6c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:02:27.767274 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:27.767244    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/582e8843-4a36-4b0d-96ca-c065aec4d6c7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "582e8843-4a36-4b0d-96ca-c065aec4d6c7" (UID: "582e8843-4a36-4b0d-96ca-c065aec4d6c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:02:27.767384 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:27.767248    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582e8843-4a36-4b0d-96ca-c065aec4d6c7-dshm" (OuterVolumeSpecName: "dshm") pod "582e8843-4a36-4b0d-96ca-c065aec4d6c7" (UID: "582e8843-4a36-4b0d-96ca-c065aec4d6c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:02:27.767384 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:27.767299    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/582e8843-4a36-4b0d-96ca-c065aec4d6c7-kube-api-access-csth6" (OuterVolumeSpecName: "kube-api-access-csth6") pod "582e8843-4a36-4b0d-96ca-c065aec4d6c7" (UID: "582e8843-4a36-4b0d-96ca-c065aec4d6c7"). InnerVolumeSpecName "kube-api-access-csth6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:02:27.865937 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:27.865895    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/582e8843-4a36-4b0d-96ca-c065aec4d6c7-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:02:27.865937 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:27.865928    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/582e8843-4a36-4b0d-96ca-c065aec4d6c7-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:02:27.865937 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:27.865939    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/582e8843-4a36-4b0d-96ca-c065aec4d6c7-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:02:27.866186 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:27.865948    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csth6\" (UniqueName: \"kubernetes.io/projected/582e8843-4a36-4b0d-96ca-c065aec4d6c7-kube-api-access-csth6\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:02:27.888898 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:27.888852    2577 generic.go:356] "Generic (PLEG): container finished" podID="582e8843-4a36-4b0d-96ca-c065aec4d6c7" containerID="3dcbee409330cd37b7506954e7bdab44bf1301e205ac6a562b6dd3d074966f9f" exitCode=0
Sep 16 18:02:27.889042 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:27.888895    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284" event={"ID":"582e8843-4a36-4b0d-96ca-c065aec4d6c7","Type":"ContainerDied","Data":"3dcbee409330cd37b7506954e7bdab44bf1301e205ac6a562b6dd3d074966f9f"}
Sep 16 18:02:27.889042 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:27.888954    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284" event={"ID":"582e8843-4a36-4b0d-96ca-c065aec4d6c7","Type":"ContainerDied","Data":"8e879e6dce29bd65506b847ecbd5a0a36e7865fe3cb925a98768816d3d9a2290"}
Sep 16 18:02:27.889042 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:27.888976    2577 scope.go:117] "RemoveContainer" containerID="3dcbee409330cd37b7506954e7bdab44bf1301e205ac6a562b6dd3d074966f9f"
Sep 16 18:02:27.889042 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:27.888906    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284"
Sep 16 18:02:27.899117 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:27.899101    2577 scope.go:117] "RemoveContainer" containerID="3dcbee409330cd37b7506954e7bdab44bf1301e205ac6a562b6dd3d074966f9f"
Sep 16 18:02:27.899365 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:02:27.899349    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dcbee409330cd37b7506954e7bdab44bf1301e205ac6a562b6dd3d074966f9f\": container with ID starting with 3dcbee409330cd37b7506954e7bdab44bf1301e205ac6a562b6dd3d074966f9f not found: ID does not exist" containerID="3dcbee409330cd37b7506954e7bdab44bf1301e205ac6a562b6dd3d074966f9f"
Sep 16 18:02:27.899413 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:27.899373    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dcbee409330cd37b7506954e7bdab44bf1301e205ac6a562b6dd3d074966f9f"} err="failed to get container status \"3dcbee409330cd37b7506954e7bdab44bf1301e205ac6a562b6dd3d074966f9f\": rpc error: code = NotFound desc = could not find container \"3dcbee409330cd37b7506954e7bdab44bf1301e205ac6a562b6dd3d074966f9f\": container with ID starting with 3dcbee409330cd37b7506954e7bdab44bf1301e205ac6a562b6dd3d074966f9f not found: ID does not exist"
Sep 16 18:02:27.921020 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:27.920993    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284"]
Sep 16 18:02:27.924604 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:27.924576    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-8z284"]
Sep 16 18:02:29.081759 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:29.081725    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="582e8843-4a36-4b0d-96ca-c065aec4d6c7" path="/var/lib/kubelet/pods/582e8843-4a36-4b0d-96ca-c065aec4d6c7/volumes"
Sep 16 18:02:32.977579 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:32.977538    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:02:36.071092 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:36.071057    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b_be8d290e-7584-426c-91fd-9e1b1e1bdad6/main/0.log"
Sep 16 18:02:38.997390 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:38.997357    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:02:45.920529 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:45.920481    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:02:46.008957 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:02:46.008922    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:03:02.228636 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.228599    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"]
Sep 16 18:03:02.237793 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:03:02.237749    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 18:03:02.237793 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:03:02.237793 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:03:02.237793 ip-10-0-136-122 kubenswrapper[2577]:  > containerID="b4b48c0d96e73e8dc17ab099613d533d679b62038301ceeee624fa4aee7d8e0a" cmd=["/bin/sleep","15"]
Sep 16 18:03:02.238126 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.237823    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v" message="" err=<
Sep 16 18:03:02.238126 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:03:02.238126 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:03:02.238126 ip-10-0-136-122 kubenswrapper[2577]:  >
Sep 16 18:03:02.238126 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:03:02.237844    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 18:03:02.238126 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:03:02.238126 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:03:02.238126 ip-10-0-136-122 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v" podUID="3c5c997d-c7d0-4366-84c1-2065e82bedcd" containerName="main" containerID="cri-o://b4b48c0d96e73e8dc17ab099613d533d679b62038301ceeee624fa4aee7d8e0a"
Sep 16 18:03:02.238126 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.237898    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v" podUID="3c5c997d-c7d0-4366-84c1-2065e82bedcd" containerName="main" containerID="cri-o://b4b48c0d96e73e8dc17ab099613d533d679b62038301ceeee624fa4aee7d8e0a" gracePeriod=60
Sep 16 18:03:02.238126 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.238106    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:03:02.255811 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.255783    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b"]
Sep 16 18:03:02.256044 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.256023    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b" podUID="be8d290e-7584-426c-91fd-9e1b1e1bdad6" containerName="main" containerID="cri-o://7633a0318eeab157082c6f062cf635302ddd052ccbe4a42b61eee68f7561e693" gracePeriod=30
Sep 16 18:03:02.394390 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.394343    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v" podUID="3c5c997d-c7d0-4366-84c1-2065e82bedcd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 16 18:03:02.413351 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.413315    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:03:02.596295 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.596265    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"
Sep 16 18:03:02.656183 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.656142    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c5c997d-c7d0-4366-84c1-2065e82bedcd-cabundle-cert\") pod \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\" (UID: \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\") "
Sep 16 18:03:02.656393 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.656197    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c5c997d-c7d0-4366-84c1-2065e82bedcd-dshm\") pod \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\" (UID: \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\") "
Sep 16 18:03:02.656393 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.656291    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c5c997d-c7d0-4366-84c1-2065e82bedcd-model-cache\") pod \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\" (UID: \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\") "
Sep 16 18:03:02.656393 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.656371    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c5c997d-c7d0-4366-84c1-2065e82bedcd-kserve-provision-location\") pod \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\" (UID: \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\") "
Sep 16 18:03:02.656555 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.656406    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c5c997d-c7d0-4366-84c1-2065e82bedcd-tls-certs\") pod \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\" (UID: \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\") "
Sep 16 18:03:02.656555 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.656425    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c5c997d-c7d0-4366-84c1-2065e82bedcd-model-cache" (OuterVolumeSpecName: "model-cache") pod "3c5c997d-c7d0-4366-84c1-2065e82bedcd" (UID: "3c5c997d-c7d0-4366-84c1-2065e82bedcd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:03:02.656555 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.656431    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c5c997d-c7d0-4366-84c1-2065e82bedcd-tmp-dir\") pod \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\" (UID: \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\") "
Sep 16 18:03:02.656555 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.656487    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzj95\" (UniqueName: \"kubernetes.io/projected/3c5c997d-c7d0-4366-84c1-2065e82bedcd-kube-api-access-kzj95\") pod \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\" (UID: \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\") "
Sep 16 18:03:02.656555 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.656504    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c5c997d-c7d0-4366-84c1-2065e82bedcd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3c5c997d-c7d0-4366-84c1-2065e82bedcd" (UID: "3c5c997d-c7d0-4366-84c1-2065e82bedcd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:03:02.656555 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.656522    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c5c997d-c7d0-4366-84c1-2065e82bedcd-home\") pod \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\" (UID: \"3c5c997d-c7d0-4366-84c1-2065e82bedcd\") "
Sep 16 18:03:02.656842 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.656647    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c5c997d-c7d0-4366-84c1-2065e82bedcd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3c5c997d-c7d0-4366-84c1-2065e82bedcd" (UID: "3c5c997d-c7d0-4366-84c1-2065e82bedcd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:03:02.656952 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.656931    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c5c997d-c7d0-4366-84c1-2065e82bedcd-cabundle-cert\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:03:02.656952 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.656942    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c5c997d-c7d0-4366-84c1-2065e82bedcd-home" (OuterVolumeSpecName: "home") pod "3c5c997d-c7d0-4366-84c1-2065e82bedcd" (UID: "3c5c997d-c7d0-4366-84c1-2065e82bedcd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:03:02.657055 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.656956    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c5c997d-c7d0-4366-84c1-2065e82bedcd-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:03:02.657055 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.656970    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c5c997d-c7d0-4366-84c1-2065e82bedcd-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:03:02.658956 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.658919    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c5c997d-c7d0-4366-84c1-2065e82bedcd-dshm" (OuterVolumeSpecName: "dshm") pod "3c5c997d-c7d0-4366-84c1-2065e82bedcd" (UID: "3c5c997d-c7d0-4366-84c1-2065e82bedcd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:03:02.659087 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.659040    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c5c997d-c7d0-4366-84c1-2065e82bedcd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3c5c997d-c7d0-4366-84c1-2065e82bedcd" (UID: "3c5c997d-c7d0-4366-84c1-2065e82bedcd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:03:02.659437 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.659408    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c5c997d-c7d0-4366-84c1-2065e82bedcd-kube-api-access-kzj95" (OuterVolumeSpecName: "kube-api-access-kzj95") pod "3c5c997d-c7d0-4366-84c1-2065e82bedcd" (UID: "3c5c997d-c7d0-4366-84c1-2065e82bedcd"). InnerVolumeSpecName "kube-api-access-kzj95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:03:02.678780 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.678748    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c5c997d-c7d0-4366-84c1-2065e82bedcd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3c5c997d-c7d0-4366-84c1-2065e82bedcd" (UID: "3c5c997d-c7d0-4366-84c1-2065e82bedcd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:03:02.687976 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.687945    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b" podUID="be8d290e-7584-426c-91fd-9e1b1e1bdad6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 16 18:03:02.758086 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.757991    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c5c997d-c7d0-4366-84c1-2065e82bedcd-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:03:02.758086 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.758028    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c5c997d-c7d0-4366-84c1-2065e82bedcd-kserve-provision-location\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:03:02.758086 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.758040    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c5c997d-c7d0-4366-84c1-2065e82bedcd-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:03:02.758086 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.758054    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzj95\" (UniqueName: \"kubernetes.io/projected/3c5c997d-c7d0-4366-84c1-2065e82bedcd-kube-api-access-kzj95\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:03:02.758086 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.758066    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c5c997d-c7d0-4366-84c1-2065e82bedcd-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:03:02.904985 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.904958    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b"
Sep 16 18:03:02.959547 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.959511    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be8d290e-7584-426c-91fd-9e1b1e1bdad6-kserve-provision-location\") pod \"be8d290e-7584-426c-91fd-9e1b1e1bdad6\" (UID: \"be8d290e-7584-426c-91fd-9e1b1e1bdad6\") "
Sep 16 18:03:02.959731 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.959572    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be8d290e-7584-426c-91fd-9e1b1e1bdad6-tmp-dir\") pod \"be8d290e-7584-426c-91fd-9e1b1e1bdad6\" (UID: \"be8d290e-7584-426c-91fd-9e1b1e1bdad6\") "
Sep 16 18:03:02.959731 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.959604    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be8d290e-7584-426c-91fd-9e1b1e1bdad6-tls-certs\") pod \"be8d290e-7584-426c-91fd-9e1b1e1bdad6\" (UID: \"be8d290e-7584-426c-91fd-9e1b1e1bdad6\") "
Sep 16 18:03:02.959731 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.959628    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be8d290e-7584-426c-91fd-9e1b1e1bdad6-cabundle-cert\") pod \"be8d290e-7584-426c-91fd-9e1b1e1bdad6\" (UID: \"be8d290e-7584-426c-91fd-9e1b1e1bdad6\") "
Sep 16 18:03:02.960067 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.960038    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8d290e-7584-426c-91fd-9e1b1e1bdad6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "be8d290e-7584-426c-91fd-9e1b1e1bdad6" (UID: "be8d290e-7584-426c-91fd-9e1b1e1bdad6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:03:02.960157 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.960084    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be8d290e-7584-426c-91fd-9e1b1e1bdad6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "be8d290e-7584-426c-91fd-9e1b1e1bdad6" (UID: "be8d290e-7584-426c-91fd-9e1b1e1bdad6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:03:02.961827 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.961805    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be8d290e-7584-426c-91fd-9e1b1e1bdad6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "be8d290e-7584-426c-91fd-9e1b1e1bdad6" (UID: "be8d290e-7584-426c-91fd-9e1b1e1bdad6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:03:02.991168 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:02.991129    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8d290e-7584-426c-91fd-9e1b1e1bdad6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "be8d290e-7584-426c-91fd-9e1b1e1bdad6" (UID: "be8d290e-7584-426c-91fd-9e1b1e1bdad6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:03:03.061214 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:03.061182    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be8d290e-7584-426c-91fd-9e1b1e1bdad6-kserve-provision-location\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:03:03.061214 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:03.061214    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be8d290e-7584-426c-91fd-9e1b1e1bdad6-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:03:03.061413 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:03.061226    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be8d290e-7584-426c-91fd-9e1b1e1bdad6-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:03:03.061413 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:03.061236    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be8d290e-7584-426c-91fd-9e1b1e1bdad6-cabundle-cert\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:03:03.087944 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:03.087910    2577 generic.go:356] "Generic (PLEG): container finished" podID="be8d290e-7584-426c-91fd-9e1b1e1bdad6" containerID="7633a0318eeab157082c6f062cf635302ddd052ccbe4a42b61eee68f7561e693" exitCode=0
Sep 16 18:03:03.088286 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:03.088254    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b" event={"ID":"be8d290e-7584-426c-91fd-9e1b1e1bdad6","Type":"ContainerDied","Data":"7633a0318eeab157082c6f062cf635302ddd052ccbe4a42b61eee68f7561e693"}
Sep 16 18:03:03.088436 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:03.088320    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b"
Sep 16 18:03:03.088436 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:03.088336    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b" event={"ID":"be8d290e-7584-426c-91fd-9e1b1e1bdad6","Type":"ContainerDied","Data":"04bfd18a14a461d5d46020443c4912a56dc3ac02b95bd8171355ab3254f39a83"}
Sep 16 18:03:03.088436 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:03.088382    2577 scope.go:117] "RemoveContainer" containerID="7633a0318eeab157082c6f062cf635302ddd052ccbe4a42b61eee68f7561e693"
Sep 16 18:03:03.091738 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:03.091714    2577 generic.go:356] "Generic (PLEG): container finished" podID="3c5c997d-c7d0-4366-84c1-2065e82bedcd" containerID="b4b48c0d96e73e8dc17ab099613d533d679b62038301ceeee624fa4aee7d8e0a" exitCode=0
Sep 16 18:03:03.091916 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:03.091755    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v" event={"ID":"3c5c997d-c7d0-4366-84c1-2065e82bedcd","Type":"ContainerDied","Data":"b4b48c0d96e73e8dc17ab099613d533d679b62038301ceeee624fa4aee7d8e0a"}
Sep 16 18:03:03.091916 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:03.091891    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v" event={"ID":"3c5c997d-c7d0-4366-84c1-2065e82bedcd","Type":"ContainerDied","Data":"3eb65d337cf881230104ea28bf5fd3ad32e40bc4fc36825e78b76edd5fdd770f"}
Sep 16 18:03:03.091916 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:03.091785    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"
Sep 16 18:03:03.101334 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:03.101309    2577 scope.go:117] "RemoveContainer" containerID="c370384188f34926b995952c34c49ecfdd4399fac0b2444d3ed7d947be81335d"
Sep 16 18:03:03.109858 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:03.109839    2577 scope.go:117] "RemoveContainer" containerID="7633a0318eeab157082c6f062cf635302ddd052ccbe4a42b61eee68f7561e693"
Sep 16 18:03:03.110159 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:03:03.110140    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7633a0318eeab157082c6f062cf635302ddd052ccbe4a42b61eee68f7561e693\": container with ID starting with 7633a0318eeab157082c6f062cf635302ddd052ccbe4a42b61eee68f7561e693 not found: ID does not exist" containerID="7633a0318eeab157082c6f062cf635302ddd052ccbe4a42b61eee68f7561e693"
Sep 16 18:03:03.110219 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:03.110167    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7633a0318eeab157082c6f062cf635302ddd052ccbe4a42b61eee68f7561e693"} err="failed to get container status \"7633a0318eeab157082c6f062cf635302ddd052ccbe4a42b61eee68f7561e693\": rpc error: code = NotFound desc = could not find container \"7633a0318eeab157082c6f062cf635302ddd052ccbe4a42b61eee68f7561e693\": container with ID starting with 7633a0318eeab157082c6f062cf635302ddd052ccbe4a42b61eee68f7561e693 not found: ID does not exist"
Sep 16 18:03:03.110219 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:03.110185    2577 scope.go:117] "RemoveContainer" containerID="c370384188f34926b995952c34c49ecfdd4399fac0b2444d3ed7d947be81335d"
Sep 16 18:03:03.110428 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:03:03.110409    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c370384188f34926b995952c34c49ecfdd4399fac0b2444d3ed7d947be81335d\": container with ID starting with c370384188f34926b995952c34c49ecfdd4399fac0b2444d3ed7d947be81335d not found: ID does not exist" containerID="c370384188f34926b995952c34c49ecfdd4399fac0b2444d3ed7d947be81335d"
Sep 16 18:03:03.110494 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:03.110439    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c370384188f34926b995952c34c49ecfdd4399fac0b2444d3ed7d947be81335d"} err="failed to get container status \"c370384188f34926b995952c34c49ecfdd4399fac0b2444d3ed7d947be81335d\": rpc error: code = NotFound desc = could not find container \"c370384188f34926b995952c34c49ecfdd4399fac0b2444d3ed7d947be81335d\": container with ID starting with c370384188f34926b995952c34c49ecfdd4399fac0b2444d3ed7d947be81335d not found: ID does not exist"
Sep 16 18:03:03.110494 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:03.110463    2577 scope.go:117] "RemoveContainer" containerID="b4b48c0d96e73e8dc17ab099613d533d679b62038301ceeee624fa4aee7d8e0a"
Sep 16 18:03:03.118919 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:03.118900    2577 scope.go:117] "RemoveContainer" containerID="9ac3ca113ee3906cc8ff7c85b66156d74a76b984e9c5c9f5211e875086ecbf78"
Sep 16 18:03:03.126703 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:03.126679    2577 scope.go:117] "RemoveContainer" containerID="b4b48c0d96e73e8dc17ab099613d533d679b62038301ceeee624fa4aee7d8e0a"
Sep 16 18:03:03.127009 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:03:03.126986    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4b48c0d96e73e8dc17ab099613d533d679b62038301ceeee624fa4aee7d8e0a\": container with ID starting with b4b48c0d96e73e8dc17ab099613d533d679b62038301ceeee624fa4aee7d8e0a not found: ID does not exist" containerID="b4b48c0d96e73e8dc17ab099613d533d679b62038301ceeee624fa4aee7d8e0a"
Sep 16 18:03:03.127099 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:03.127012    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4b48c0d96e73e8dc17ab099613d533d679b62038301ceeee624fa4aee7d8e0a"} err="failed to get container status \"b4b48c0d96e73e8dc17ab099613d533d679b62038301ceeee624fa4aee7d8e0a\": rpc error: code = NotFound desc = could not find container \"b4b48c0d96e73e8dc17ab099613d533d679b62038301ceeee624fa4aee7d8e0a\": container with ID starting with b4b48c0d96e73e8dc17ab099613d533d679b62038301ceeee624fa4aee7d8e0a not found: ID does not exist"
Sep 16 18:03:03.127099 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:03.127030    2577 scope.go:117] "RemoveContainer" containerID="9ac3ca113ee3906cc8ff7c85b66156d74a76b984e9c5c9f5211e875086ecbf78"
Sep 16 18:03:03.127262 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:03:03.127242    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ac3ca113ee3906cc8ff7c85b66156d74a76b984e9c5c9f5211e875086ecbf78\": container with ID starting with 9ac3ca113ee3906cc8ff7c85b66156d74a76b984e9c5c9f5211e875086ecbf78 not found: ID does not exist" containerID="9ac3ca113ee3906cc8ff7c85b66156d74a76b984e9c5c9f5211e875086ecbf78"
Sep 16 18:03:03.127331 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:03.127267    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ac3ca113ee3906cc8ff7c85b66156d74a76b984e9c5c9f5211e875086ecbf78"} err="failed to get container status \"9ac3ca113ee3906cc8ff7c85b66156d74a76b984e9c5c9f5211e875086ecbf78\": rpc error: code = NotFound desc = could not find container \"9ac3ca113ee3906cc8ff7c85b66156d74a76b984e9c5c9f5211e875086ecbf78\": container with ID starting with 9ac3ca113ee3906cc8ff7c85b66156d74a76b984e9c5c9f5211e875086ecbf78 not found: ID does not exist"
Sep 16 18:03:03.154632 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:03.154597    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"]
Sep 16 18:03:03.164493 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:03.164466    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-fvp8v"]
Sep 16 18:03:03.187173 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:03.187145    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b"]
Sep 16 18:03:03.198265 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:03.198236    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b5d76dd49-mxg8b"]
Sep 16 18:03:05.080975 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:05.080948    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c5c997d-c7d0-4366-84c1-2065e82bedcd" path="/var/lib/kubelet/pods/3c5c997d-c7d0-4366-84c1-2065e82bedcd/volumes"
Sep 16 18:03:05.081374 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:05.081361    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be8d290e-7584-426c-91fd-9e1b1e1bdad6" path="/var/lib/kubelet/pods/be8d290e-7584-426c-91fd-9e1b1e1bdad6/volumes"
Sep 16 18:03:05.729685 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:05.729646    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:03:05.972385 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:05.972352    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:03:06.948291 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:06.948251    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:03:10.889118 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:10.889079    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:03:12.820549 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:12.820519    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:03:13.974946 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:13.974912    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:03:28.258071 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:28.258018    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:03:50.601615 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:50.601576    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:03:52.100162 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:52.100124    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:03:58.450778 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:58.450739    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:03:58.709392 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:03:58.709306    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:04:00.080965 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:00.080925    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:04:06.873121 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:06.873079    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:04:06.974748 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:06.974708    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:04:11.641562 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:11.641525    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:04:54.708841 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.708798    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"]
Sep 16 18:04:54.709480 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.709320    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be8d290e-7584-426c-91fd-9e1b1e1bdad6" containerName="storage-initializer"
Sep 16 18:04:54.709480 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.709340    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="be8d290e-7584-426c-91fd-9e1b1e1bdad6" containerName="storage-initializer"
Sep 16 18:04:54.709480 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.709359    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="582e8843-4a36-4b0d-96ca-c065aec4d6c7" containerName="main"
Sep 16 18:04:54.709480 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.709368    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="582e8843-4a36-4b0d-96ca-c065aec4d6c7" containerName="main"
Sep 16 18:04:54.709480 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.709386    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be8d290e-7584-426c-91fd-9e1b1e1bdad6" containerName="main"
Sep 16 18:04:54.709480 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.709394    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="be8d290e-7584-426c-91fd-9e1b1e1bdad6" containerName="main"
Sep 16 18:04:54.709480 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.709409    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c5c997d-c7d0-4366-84c1-2065e82bedcd" containerName="storage-initializer"
Sep 16 18:04:54.709480 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.709419    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c5c997d-c7d0-4366-84c1-2065e82bedcd" containerName="storage-initializer"
Sep 16 18:04:54.709480 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.709433    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c5c997d-c7d0-4366-84c1-2065e82bedcd" containerName="main"
Sep 16 18:04:54.709480 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.709442    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c5c997d-c7d0-4366-84c1-2065e82bedcd" containerName="main"
Sep 16 18:04:54.710029 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.709516    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="582e8843-4a36-4b0d-96ca-c065aec4d6c7" containerName="main"
Sep 16 18:04:54.710029 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.709531    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c5c997d-c7d0-4366-84c1-2065e82bedcd" containerName="main"
Sep 16 18:04:54.710029 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.709540    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="be8d290e-7584-426c-91fd-9e1b1e1bdad6" containerName="main"
Sep 16 18:04:54.712918 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.712896    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"
Sep 16 18:04:54.716243 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.716218    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 16 18:04:54.717651 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.717633    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 16 18:04:54.717749 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.717726    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:04:54.717818 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.717737    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 16 18:04:54.718923 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.718904    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-rhvtt\""
Sep 16 18:04:54.719017 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.718941    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 16 18:04:54.736244 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.736210    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"]
Sep 16 18:04:54.803459 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.803414    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955d31df-891d-49f3-bb85-728f61bcfb5a-home\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj\" (UID: \"955d31df-891d-49f3-bb85-728f61bcfb5a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"
Sep 16 18:04:54.803459 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.803463    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955d31df-891d-49f3-bb85-728f61bcfb5a-model-cache\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj\" (UID: \"955d31df-891d-49f3-bb85-728f61bcfb5a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"
Sep 16 18:04:54.803722 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.803483    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955d31df-891d-49f3-bb85-728f61bcfb5a-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj\" (UID: \"955d31df-891d-49f3-bb85-728f61bcfb5a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"
Sep 16 18:04:54.803722 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.803499    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955d31df-891d-49f3-bb85-728f61bcfb5a-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj\" (UID: \"955d31df-891d-49f3-bb85-728f61bcfb5a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"
Sep 16 18:04:54.803722 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.803569    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/955d31df-891d-49f3-bb85-728f61bcfb5a-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj\" (UID: \"955d31df-891d-49f3-bb85-728f61bcfb5a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"
Sep 16 18:04:54.803722 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.803628    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f84xj\" (UniqueName: \"kubernetes.io/projected/955d31df-891d-49f3-bb85-728f61bcfb5a-kube-api-access-f84xj\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj\" (UID: \"955d31df-891d-49f3-bb85-728f61bcfb5a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"
Sep 16 18:04:54.803722 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.803694    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/955d31df-891d-49f3-bb85-728f61bcfb5a-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj\" (UID: \"955d31df-891d-49f3-bb85-728f61bcfb5a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"
Sep 16 18:04:54.804025 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.803737    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955d31df-891d-49f3-bb85-728f61bcfb5a-dshm\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj\" (UID: \"955d31df-891d-49f3-bb85-728f61bcfb5a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"
Sep 16 18:04:54.892117 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.892081    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:04:54.904982 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.904952    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f84xj\" (UniqueName: \"kubernetes.io/projected/955d31df-891d-49f3-bb85-728f61bcfb5a-kube-api-access-f84xj\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj\" (UID: \"955d31df-891d-49f3-bb85-728f61bcfb5a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"
Sep 16 18:04:54.905143 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.904993    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/955d31df-891d-49f3-bb85-728f61bcfb5a-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj\" (UID: \"955d31df-891d-49f3-bb85-728f61bcfb5a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"
Sep 16 18:04:54.905143 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.905036    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955d31df-891d-49f3-bb85-728f61bcfb5a-dshm\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj\" (UID: \"955d31df-891d-49f3-bb85-728f61bcfb5a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"
Sep 16 18:04:54.905143 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.905080    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955d31df-891d-49f3-bb85-728f61bcfb5a-home\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj\" (UID: \"955d31df-891d-49f3-bb85-728f61bcfb5a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"
Sep 16 18:04:54.905143 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.905114    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955d31df-891d-49f3-bb85-728f61bcfb5a-model-cache\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj\" (UID: \"955d31df-891d-49f3-bb85-728f61bcfb5a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"
Sep 16 18:04:54.905364 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.905144    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955d31df-891d-49f3-bb85-728f61bcfb5a-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj\" (UID: \"955d31df-891d-49f3-bb85-728f61bcfb5a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"
Sep 16 18:04:54.905364 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.905169    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955d31df-891d-49f3-bb85-728f61bcfb5a-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj\" (UID: \"955d31df-891d-49f3-bb85-728f61bcfb5a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"
Sep 16 18:04:54.905364 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.905204    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/955d31df-891d-49f3-bb85-728f61bcfb5a-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj\" (UID: \"955d31df-891d-49f3-bb85-728f61bcfb5a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"
Sep 16 18:04:54.905516 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.905486    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955d31df-891d-49f3-bb85-728f61bcfb5a-home\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj\" (UID: \"955d31df-891d-49f3-bb85-728f61bcfb5a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"
Sep 16 18:04:54.905516 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.905496    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/955d31df-891d-49f3-bb85-728f61bcfb5a-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj\" (UID: \"955d31df-891d-49f3-bb85-728f61bcfb5a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"
Sep 16 18:04:54.905634 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.905573    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955d31df-891d-49f3-bb85-728f61bcfb5a-model-cache\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj\" (UID: \"955d31df-891d-49f3-bb85-728f61bcfb5a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"
Sep 16 18:04:54.905634 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.905625    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955d31df-891d-49f3-bb85-728f61bcfb5a-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj\" (UID: \"955d31df-891d-49f3-bb85-728f61bcfb5a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"
Sep 16 18:04:54.906003 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.905981    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/955d31df-891d-49f3-bb85-728f61bcfb5a-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj\" (UID: \"955d31df-891d-49f3-bb85-728f61bcfb5a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"
Sep 16 18:04:54.907432 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.907408    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955d31df-891d-49f3-bb85-728f61bcfb5a-dshm\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj\" (UID: \"955d31df-891d-49f3-bb85-728f61bcfb5a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"
Sep 16 18:04:54.907666 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.907646    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955d31df-891d-49f3-bb85-728f61bcfb5a-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj\" (UID: \"955d31df-891d-49f3-bb85-728f61bcfb5a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"
Sep 16 18:04:54.925638 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:54.925612    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f84xj\" (UniqueName: \"kubernetes.io/projected/955d31df-891d-49f3-bb85-728f61bcfb5a-kube-api-access-f84xj\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj\" (UID: \"955d31df-891d-49f3-bb85-728f61bcfb5a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"
Sep 16 18:04:55.023656 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:55.023565    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"
Sep 16 18:04:55.168935 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:55.168894    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz"]
Sep 16 18:04:55.172661 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:55.172642    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz"
Sep 16 18:04:55.195251 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:55.195216    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz"]
Sep 16 18:04:55.208471 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:55.208431    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d0f636-c265-40a2-be7d-3085e2e62564-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz\" (UID: \"f2d0f636-c265-40a2-be7d-3085e2e62564\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz"
Sep 16 18:04:55.208624 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:55.208568    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d0f636-c265-40a2-be7d-3085e2e62564-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz\" (UID: \"f2d0f636-c265-40a2-be7d-3085e2e62564\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz"
Sep 16 18:04:55.208705 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:55.208629    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2d0f636-c265-40a2-be7d-3085e2e62564-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz\" (UID: \"f2d0f636-c265-40a2-be7d-3085e2e62564\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz"
Sep 16 18:04:55.208763 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:55.208719    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2d0f636-c265-40a2-be7d-3085e2e62564-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz\" (UID: \"f2d0f636-c265-40a2-be7d-3085e2e62564\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz"
Sep 16 18:04:55.227547 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:55.227509    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:04:55.315727 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:55.315691    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d0f636-c265-40a2-be7d-3085e2e62564-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz\" (UID: \"f2d0f636-c265-40a2-be7d-3085e2e62564\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz"
Sep 16 18:04:55.316023 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:55.315749    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2d0f636-c265-40a2-be7d-3085e2e62564-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz\" (UID: \"f2d0f636-c265-40a2-be7d-3085e2e62564\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz"
Sep 16 18:04:55.316023 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:55.315790    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2d0f636-c265-40a2-be7d-3085e2e62564-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz\" (UID: \"f2d0f636-c265-40a2-be7d-3085e2e62564\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz"
Sep 16 18:04:55.316023 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:55.315838    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d0f636-c265-40a2-be7d-3085e2e62564-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz\" (UID: \"f2d0f636-c265-40a2-be7d-3085e2e62564\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz"
Sep 16 18:04:55.316334 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:55.316313    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2d0f636-c265-40a2-be7d-3085e2e62564-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz\" (UID: \"f2d0f636-c265-40a2-be7d-3085e2e62564\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz"
Sep 16 18:04:55.316406 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:55.316368    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d0f636-c265-40a2-be7d-3085e2e62564-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz\" (UID: \"f2d0f636-c265-40a2-be7d-3085e2e62564\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz"
Sep 16 18:04:55.316616 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:55.316592    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2d0f636-c265-40a2-be7d-3085e2e62564-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz\" (UID: \"f2d0f636-c265-40a2-be7d-3085e2e62564\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz"
Sep 16 18:04:55.318346 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:55.318325    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d0f636-c265-40a2-be7d-3085e2e62564-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz\" (UID: \"f2d0f636-c265-40a2-be7d-3085e2e62564\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz"
Sep 16 18:04:55.322995 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:55.322975    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"]
Sep 16 18:04:55.325241 ip-10-0-136-122 kubenswrapper[2577]: W0916 18:04:55.325203    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod955d31df_891d_49f3_bb85_728f61bcfb5a.slice/crio-07e2a3759d583792b86fa4f1ef9fde0cc13ed93811062e3ed2ff47be082b3e2e WatchSource:0}: Error finding container 07e2a3759d583792b86fa4f1ef9fde0cc13ed93811062e3ed2ff47be082b3e2e: Status 404 returned error can't find the container with id 07e2a3759d583792b86fa4f1ef9fde0cc13ed93811062e3ed2ff47be082b3e2e
Sep 16 18:04:55.362703 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:55.362669    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:04:55.484544 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:55.484498    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz"
Sep 16 18:04:55.689198 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:55.689079    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj" event={"ID":"955d31df-891d-49f3-bb85-728f61bcfb5a","Type":"ContainerStarted","Data":"b2dd1701ab940bb1bfcc8a494c2da4665f36f571b8a8f1f1b22b0fea2103c8bd"}
Sep 16 18:04:55.689198 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:55.689152    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj" event={"ID":"955d31df-891d-49f3-bb85-728f61bcfb5a","Type":"ContainerStarted","Data":"07e2a3759d583792b86fa4f1ef9fde0cc13ed93811062e3ed2ff47be082b3e2e"}
Sep 16 18:04:55.768984 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:55.768958    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz"]
Sep 16 18:04:55.771063 ip-10-0-136-122 kubenswrapper[2577]: W0916 18:04:55.771034    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d0f636_c265_40a2_be7d_3085e2e62564.slice/crio-0ccf3a1668975ea1be977b02b4ae1cfb5d1f7a1a73949ffd9172acb1801975d1 WatchSource:0}: Error finding container 0ccf3a1668975ea1be977b02b4ae1cfb5d1f7a1a73949ffd9172acb1801975d1: Status 404 returned error can't find the container with id 0ccf3a1668975ea1be977b02b4ae1cfb5d1f7a1a73949ffd9172acb1801975d1
Sep 16 18:04:56.700001 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:56.699940    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz" event={"ID":"f2d0f636-c265-40a2-be7d-3085e2e62564","Type":"ContainerStarted","Data":"b9556aacdbfe42951b1cd4063c4125571b27916ee146e1f94f981de794adca2b"}
Sep 16 18:04:56.700199 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:56.700046    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz" event={"ID":"f2d0f636-c265-40a2-be7d-3085e2e62564","Type":"ContainerStarted","Data":"0ccf3a1668975ea1be977b02b4ae1cfb5d1f7a1a73949ffd9172acb1801975d1"}
Sep 16 18:04:58.714237 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:58.714142    2577 generic.go:356] "Generic (PLEG): container finished" podID="955d31df-891d-49f3-bb85-728f61bcfb5a" containerID="b2dd1701ab940bb1bfcc8a494c2da4665f36f571b8a8f1f1b22b0fea2103c8bd" exitCode=0
Sep 16 18:04:58.714237 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:58.714214    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj" event={"ID":"955d31df-891d-49f3-bb85-728f61bcfb5a","Type":"ContainerDied","Data":"b2dd1701ab940bb1bfcc8a494c2da4665f36f571b8a8f1f1b22b0fea2103c8bd"}
Sep 16 18:04:58.715776 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:58.715754    2577 generic.go:356] "Generic (PLEG): container finished" podID="f2d0f636-c265-40a2-be7d-3085e2e62564" containerID="b9556aacdbfe42951b1cd4063c4125571b27916ee146e1f94f981de794adca2b" exitCode=0
Sep 16 18:04:58.715917 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:58.715803    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz" event={"ID":"f2d0f636-c265-40a2-be7d-3085e2e62564","Type":"ContainerDied","Data":"b9556aacdbfe42951b1cd4063c4125571b27916ee146e1f94f981de794adca2b"}
Sep 16 18:04:59.726946 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:59.726908    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj" event={"ID":"955d31df-891d-49f3-bb85-728f61bcfb5a","Type":"ContainerStarted","Data":"caec902d2f5ab7cbe183e71353a01ea07bb915a6689bc9fd0e5f2aa2a70a2eb2"}
Sep 16 18:04:59.728987 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:59.728958    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz" event={"ID":"f2d0f636-c265-40a2-be7d-3085e2e62564","Type":"ContainerStarted","Data":"bfdcdf22374da38b0beb7b0db3e250fe20fb70c99c82d6f2c99aea14fb70f1bf"}
Sep 16 18:04:59.729192 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:59.729166    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz"
Sep 16 18:04:59.778211 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:59.778148    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj" podStartSLOduration=3.249505922 podStartE2EDuration="5.778124351s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.528618429s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:04:59.777188562 +0000 UTC m=+2421.139867348" watchObservedRunningTime="2026-09-16 18:04:59.778124351 +0000 UTC m=+2421.140803137"
Sep 16 18:04:59.820580 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:04:59.820507    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz" podStartSLOduration=2.310833891 podStartE2EDuration="4.820483186s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.509649295s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:04:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:04:59.81368893 +0000 UTC m=+2421.176367717" watchObservedRunningTime="2026-09-16 18:04:59.820483186 +0000 UTC m=+2421.183161974"
Sep 16 18:05:00.024257 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:00.024158    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"
Sep 16 18:05:05.024228 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:05.024128    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"
Sep 16 18:05:05.037926 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:05.037889    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"
Sep 16 18:05:05.777233 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:05.777200    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"
Sep 16 18:05:15.800761 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:15.800719    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:05:24.182525 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:24.182468    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:05:30.747547 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:30.747507    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz"
Sep 16 18:05:35.982833 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:35.982793    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz_f2d0f636-c265-40a2-be7d-3085e2e62564/main/0.log"
Sep 16 18:05:39.503999 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.503956    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:05:39.527696 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.527665    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"]
Sep 16 18:05:39.537438 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:05:39.537404    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 18:05:39.537438 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:05:39.537438 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:05:39.537438 ip-10-0-136-122 kubenswrapper[2577]:  > containerID="caec902d2f5ab7cbe183e71353a01ea07bb915a6689bc9fd0e5f2aa2a70a2eb2" cmd=["/bin/sleep","15"]
Sep 16 18:05:39.537897 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.537452    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj" message="" err=<
Sep 16 18:05:39.537897 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:05:39.537897 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:05:39.537897 ip-10-0-136-122 kubenswrapper[2577]:  >
Sep 16 18:05:39.537897 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:05:39.537467    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 18:05:39.537897 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:05:39.537897 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:05:39.537897 ip-10-0-136-122 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj" podUID="955d31df-891d-49f3-bb85-728f61bcfb5a" containerName="main" containerID="cri-o://caec902d2f5ab7cbe183e71353a01ea07bb915a6689bc9fd0e5f2aa2a70a2eb2"
Sep 16 18:05:39.537897 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.537500    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj" podUID="955d31df-891d-49f3-bb85-728f61bcfb5a" containerName="main" containerID="cri-o://caec902d2f5ab7cbe183e71353a01ea07bb915a6689bc9fd0e5f2aa2a70a2eb2" gracePeriod=60
Sep 16 18:05:39.542668 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.542640    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz"]
Sep 16 18:05:39.542932 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.542909    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz" podUID="f2d0f636-c265-40a2-be7d-3085e2e62564" containerName="main" containerID="cri-o://bfdcdf22374da38b0beb7b0db3e250fe20fb70c99c82d6f2c99aea14fb70f1bf" gracePeriod=30
Sep 16 18:05:39.641671 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.641628    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:05:39.765874 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.765820    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj" podUID="955d31df-891d-49f3-bb85-728f61bcfb5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Sep 16 18:05:39.902562 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.902530    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"
Sep 16 18:05:39.913222 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.913194    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f84xj\" (UniqueName: \"kubernetes.io/projected/955d31df-891d-49f3-bb85-728f61bcfb5a-kube-api-access-f84xj\") pod \"955d31df-891d-49f3-bb85-728f61bcfb5a\" (UID: \"955d31df-891d-49f3-bb85-728f61bcfb5a\") "
Sep 16 18:05:39.913398 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.913240    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955d31df-891d-49f3-bb85-728f61bcfb5a-model-cache\") pod \"955d31df-891d-49f3-bb85-728f61bcfb5a\" (UID: \"955d31df-891d-49f3-bb85-728f61bcfb5a\") "
Sep 16 18:05:39.913398 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.913266    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955d31df-891d-49f3-bb85-728f61bcfb5a-home\") pod \"955d31df-891d-49f3-bb85-728f61bcfb5a\" (UID: \"955d31df-891d-49f3-bb85-728f61bcfb5a\") "
Sep 16 18:05:39.913398 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.913290    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/955d31df-891d-49f3-bb85-728f61bcfb5a-cabundle-cert\") pod \"955d31df-891d-49f3-bb85-728f61bcfb5a\" (UID: \"955d31df-891d-49f3-bb85-728f61bcfb5a\") "
Sep 16 18:05:39.913590 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.913394    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/955d31df-891d-49f3-bb85-728f61bcfb5a-kserve-provision-location\") pod \"955d31df-891d-49f3-bb85-728f61bcfb5a\" (UID: \"955d31df-891d-49f3-bb85-728f61bcfb5a\") "
Sep 16 18:05:39.913590 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.913446    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955d31df-891d-49f3-bb85-728f61bcfb5a-tmp-dir\") pod \"955d31df-891d-49f3-bb85-728f61bcfb5a\" (UID: \"955d31df-891d-49f3-bb85-728f61bcfb5a\") "
Sep 16 18:05:39.913590 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.913491    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955d31df-891d-49f3-bb85-728f61bcfb5a-tls-certs\") pod \"955d31df-891d-49f3-bb85-728f61bcfb5a\" (UID: \"955d31df-891d-49f3-bb85-728f61bcfb5a\") "
Sep 16 18:05:39.913590 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.913521    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955d31df-891d-49f3-bb85-728f61bcfb5a-dshm\") pod \"955d31df-891d-49f3-bb85-728f61bcfb5a\" (UID: \"955d31df-891d-49f3-bb85-728f61bcfb5a\") "
Sep 16 18:05:39.913590 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.913502    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955d31df-891d-49f3-bb85-728f61bcfb5a-model-cache" (OuterVolumeSpecName: "model-cache") pod "955d31df-891d-49f3-bb85-728f61bcfb5a" (UID: "955d31df-891d-49f3-bb85-728f61bcfb5a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:05:39.913590 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.913567    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/955d31df-891d-49f3-bb85-728f61bcfb5a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "955d31df-891d-49f3-bb85-728f61bcfb5a" (UID: "955d31df-891d-49f3-bb85-728f61bcfb5a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:05:39.913590 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.913580    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955d31df-891d-49f3-bb85-728f61bcfb5a-home" (OuterVolumeSpecName: "home") pod "955d31df-891d-49f3-bb85-728f61bcfb5a" (UID: "955d31df-891d-49f3-bb85-728f61bcfb5a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:05:39.913975 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.913731    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955d31df-891d-49f3-bb85-728f61bcfb5a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "955d31df-891d-49f3-bb85-728f61bcfb5a" (UID: "955d31df-891d-49f3-bb85-728f61bcfb5a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:05:39.913975 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.913903    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/955d31df-891d-49f3-bb85-728f61bcfb5a-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:05:39.913975 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.913931    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/955d31df-891d-49f3-bb85-728f61bcfb5a-cabundle-cert\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:05:39.913975 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.913946    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955d31df-891d-49f3-bb85-728f61bcfb5a-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:05:39.913975 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.913957    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/955d31df-891d-49f3-bb85-728f61bcfb5a-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:05:39.915723 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.915698    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/955d31df-891d-49f3-bb85-728f61bcfb5a-kube-api-access-f84xj" (OuterVolumeSpecName: "kube-api-access-f84xj") pod "955d31df-891d-49f3-bb85-728f61bcfb5a" (UID: "955d31df-891d-49f3-bb85-728f61bcfb5a"). InnerVolumeSpecName "kube-api-access-f84xj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:05:39.915834 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.915775    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955d31df-891d-49f3-bb85-728f61bcfb5a-dshm" (OuterVolumeSpecName: "dshm") pod "955d31df-891d-49f3-bb85-728f61bcfb5a" (UID: "955d31df-891d-49f3-bb85-728f61bcfb5a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:05:39.916016 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.915996    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/955d31df-891d-49f3-bb85-728f61bcfb5a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "955d31df-891d-49f3-bb85-728f61bcfb5a" (UID: "955d31df-891d-49f3-bb85-728f61bcfb5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:05:39.936190 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.936157    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955d31df-891d-49f3-bb85-728f61bcfb5a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "955d31df-891d-49f3-bb85-728f61bcfb5a" (UID: "955d31df-891d-49f3-bb85-728f61bcfb5a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:05:39.959901 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.959853    2577 generic.go:356] "Generic (PLEG): container finished" podID="955d31df-891d-49f3-bb85-728f61bcfb5a" containerID="caec902d2f5ab7cbe183e71353a01ea07bb915a6689bc9fd0e5f2aa2a70a2eb2" exitCode=0
Sep 16 18:05:39.960022 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.959918    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj" event={"ID":"955d31df-891d-49f3-bb85-728f61bcfb5a","Type":"ContainerDied","Data":"caec902d2f5ab7cbe183e71353a01ea07bb915a6689bc9fd0e5f2aa2a70a2eb2"}
Sep 16 18:05:39.960022 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.959954    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"
Sep 16 18:05:39.960022 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.959981    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj" event={"ID":"955d31df-891d-49f3-bb85-728f61bcfb5a","Type":"ContainerDied","Data":"07e2a3759d583792b86fa4f1ef9fde0cc13ed93811062e3ed2ff47be082b3e2e"}
Sep 16 18:05:39.960022 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:39.960004    2577 scope.go:117] "RemoveContainer" containerID="caec902d2f5ab7cbe183e71353a01ea07bb915a6689bc9fd0e5f2aa2a70a2eb2"
Sep 16 18:05:40.010391 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.010359    2577 scope.go:117] "RemoveContainer" containerID="b2dd1701ab940bb1bfcc8a494c2da4665f36f571b8a8f1f1b22b0fea2103c8bd"
Sep 16 18:05:40.015076 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.015046    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f84xj\" (UniqueName: \"kubernetes.io/projected/955d31df-891d-49f3-bb85-728f61bcfb5a-kube-api-access-f84xj\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:05:40.015181 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.015079    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/955d31df-891d-49f3-bb85-728f61bcfb5a-kserve-provision-location\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:05:40.015181 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.015098    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955d31df-891d-49f3-bb85-728f61bcfb5a-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:05:40.015181 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.015112    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/955d31df-891d-49f3-bb85-728f61bcfb5a-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:05:40.019320 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.019299    2577 scope.go:117] "RemoveContainer" containerID="caec902d2f5ab7cbe183e71353a01ea07bb915a6689bc9fd0e5f2aa2a70a2eb2"
Sep 16 18:05:40.019685 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:05:40.019662    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caec902d2f5ab7cbe183e71353a01ea07bb915a6689bc9fd0e5f2aa2a70a2eb2\": container with ID starting with caec902d2f5ab7cbe183e71353a01ea07bb915a6689bc9fd0e5f2aa2a70a2eb2 not found: ID does not exist" containerID="caec902d2f5ab7cbe183e71353a01ea07bb915a6689bc9fd0e5f2aa2a70a2eb2"
Sep 16 18:05:40.019770 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.019693    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caec902d2f5ab7cbe183e71353a01ea07bb915a6689bc9fd0e5f2aa2a70a2eb2"} err="failed to get container status \"caec902d2f5ab7cbe183e71353a01ea07bb915a6689bc9fd0e5f2aa2a70a2eb2\": rpc error: code = NotFound desc = could not find container \"caec902d2f5ab7cbe183e71353a01ea07bb915a6689bc9fd0e5f2aa2a70a2eb2\": container with ID starting with caec902d2f5ab7cbe183e71353a01ea07bb915a6689bc9fd0e5f2aa2a70a2eb2 not found: ID does not exist"
Sep 16 18:05:40.019770 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.019712    2577 scope.go:117] "RemoveContainer" containerID="b2dd1701ab940bb1bfcc8a494c2da4665f36f571b8a8f1f1b22b0fea2103c8bd"
Sep 16 18:05:40.020080 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:05:40.020059    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2dd1701ab940bb1bfcc8a494c2da4665f36f571b8a8f1f1b22b0fea2103c8bd\": container with ID starting with b2dd1701ab940bb1bfcc8a494c2da4665f36f571b8a8f1f1b22b0fea2103c8bd not found: ID does not exist" containerID="b2dd1701ab940bb1bfcc8a494c2da4665f36f571b8a8f1f1b22b0fea2103c8bd"
Sep 16 18:05:40.020151 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.020091    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2dd1701ab940bb1bfcc8a494c2da4665f36f571b8a8f1f1b22b0fea2103c8bd"} err="failed to get container status \"b2dd1701ab940bb1bfcc8a494c2da4665f36f571b8a8f1f1b22b0fea2103c8bd\": rpc error: code = NotFound desc = could not find container \"b2dd1701ab940bb1bfcc8a494c2da4665f36f571b8a8f1f1b22b0fea2103c8bd\": container with ID starting with b2dd1701ab940bb1bfcc8a494c2da4665f36f571b8a8f1f1b22b0fea2103c8bd not found: ID does not exist"
Sep 16 18:05:40.034988 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.034957    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"]
Sep 16 18:05:40.040840 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.040812    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-9b2zj"]
Sep 16 18:05:40.128795 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.128769    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz"
Sep 16 18:05:40.216714 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.216680    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d0f636-c265-40a2-be7d-3085e2e62564-tls-certs\") pod \"f2d0f636-c265-40a2-be7d-3085e2e62564\" (UID: \"f2d0f636-c265-40a2-be7d-3085e2e62564\") "
Sep 16 18:05:40.216714 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.216717    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2d0f636-c265-40a2-be7d-3085e2e62564-kserve-provision-location\") pod \"f2d0f636-c265-40a2-be7d-3085e2e62564\" (UID: \"f2d0f636-c265-40a2-be7d-3085e2e62564\") "
Sep 16 18:05:40.216952 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.216742    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d0f636-c265-40a2-be7d-3085e2e62564-tmp-dir\") pod \"f2d0f636-c265-40a2-be7d-3085e2e62564\" (UID: \"f2d0f636-c265-40a2-be7d-3085e2e62564\") "
Sep 16 18:05:40.216952 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.216775    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2d0f636-c265-40a2-be7d-3085e2e62564-cabundle-cert\") pod \"f2d0f636-c265-40a2-be7d-3085e2e62564\" (UID: \"f2d0f636-c265-40a2-be7d-3085e2e62564\") "
Sep 16 18:05:40.217223 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.217197    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d0f636-c265-40a2-be7d-3085e2e62564-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f2d0f636-c265-40a2-be7d-3085e2e62564" (UID: "f2d0f636-c265-40a2-be7d-3085e2e62564"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:05:40.217290 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.217237    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2d0f636-c265-40a2-be7d-3085e2e62564-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f2d0f636-c265-40a2-be7d-3085e2e62564" (UID: "f2d0f636-c265-40a2-be7d-3085e2e62564"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:05:40.219119 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.219093    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2d0f636-c265-40a2-be7d-3085e2e62564-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f2d0f636-c265-40a2-be7d-3085e2e62564" (UID: "f2d0f636-c265-40a2-be7d-3085e2e62564"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:05:40.240072 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.240015    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d0f636-c265-40a2-be7d-3085e2e62564-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f2d0f636-c265-40a2-be7d-3085e2e62564" (UID: "f2d0f636-c265-40a2-be7d-3085e2e62564"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:05:40.317771 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.317736    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2d0f636-c265-40a2-be7d-3085e2e62564-cabundle-cert\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:05:40.317771 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.317766    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d0f636-c265-40a2-be7d-3085e2e62564-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:05:40.317771 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.317776    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2d0f636-c265-40a2-be7d-3085e2e62564-kserve-provision-location\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:05:40.318039 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.317785    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d0f636-c265-40a2-be7d-3085e2e62564-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:05:40.971143 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.971047    2577 generic.go:356] "Generic (PLEG): container finished" podID="f2d0f636-c265-40a2-be7d-3085e2e62564" containerID="bfdcdf22374da38b0beb7b0db3e250fe20fb70c99c82d6f2c99aea14fb70f1bf" exitCode=0
Sep 16 18:05:40.971143 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.971117    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz"
Sep 16 18:05:40.971622 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.971156    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz" event={"ID":"f2d0f636-c265-40a2-be7d-3085e2e62564","Type":"ContainerDied","Data":"bfdcdf22374da38b0beb7b0db3e250fe20fb70c99c82d6f2c99aea14fb70f1bf"}
Sep 16 18:05:40.971622 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.971216    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz" event={"ID":"f2d0f636-c265-40a2-be7d-3085e2e62564","Type":"ContainerDied","Data":"0ccf3a1668975ea1be977b02b4ae1cfb5d1f7a1a73949ffd9172acb1801975d1"}
Sep 16 18:05:40.971622 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.971248    2577 scope.go:117] "RemoveContainer" containerID="bfdcdf22374da38b0beb7b0db3e250fe20fb70c99c82d6f2c99aea14fb70f1bf"
Sep 16 18:05:40.982008 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.981942    2577 scope.go:117] "RemoveContainer" containerID="b9556aacdbfe42951b1cd4063c4125571b27916ee146e1f94f981de794adca2b"
Sep 16 18:05:40.991452 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.991427    2577 scope.go:117] "RemoveContainer" containerID="bfdcdf22374da38b0beb7b0db3e250fe20fb70c99c82d6f2c99aea14fb70f1bf"
Sep 16 18:05:40.991813 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:05:40.991790    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfdcdf22374da38b0beb7b0db3e250fe20fb70c99c82d6f2c99aea14fb70f1bf\": container with ID starting with bfdcdf22374da38b0beb7b0db3e250fe20fb70c99c82d6f2c99aea14fb70f1bf not found: ID does not exist" containerID="bfdcdf22374da38b0beb7b0db3e250fe20fb70c99c82d6f2c99aea14fb70f1bf"
Sep 16 18:05:40.991931 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.991826    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfdcdf22374da38b0beb7b0db3e250fe20fb70c99c82d6f2c99aea14fb70f1bf"} err="failed to get container status \"bfdcdf22374da38b0beb7b0db3e250fe20fb70c99c82d6f2c99aea14fb70f1bf\": rpc error: code = NotFound desc = could not find container \"bfdcdf22374da38b0beb7b0db3e250fe20fb70c99c82d6f2c99aea14fb70f1bf\": container with ID starting with bfdcdf22374da38b0beb7b0db3e250fe20fb70c99c82d6f2c99aea14fb70f1bf not found: ID does not exist"
Sep 16 18:05:40.991931 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.991850    2577 scope.go:117] "RemoveContainer" containerID="b9556aacdbfe42951b1cd4063c4125571b27916ee146e1f94f981de794adca2b"
Sep 16 18:05:40.992229 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:05:40.992210    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9556aacdbfe42951b1cd4063c4125571b27916ee146e1f94f981de794adca2b\": container with ID starting with b9556aacdbfe42951b1cd4063c4125571b27916ee146e1f94f981de794adca2b not found: ID does not exist" containerID="b9556aacdbfe42951b1cd4063c4125571b27916ee146e1f94f981de794adca2b"
Sep 16 18:05:40.992296 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:40.992234    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9556aacdbfe42951b1cd4063c4125571b27916ee146e1f94f981de794adca2b"} err="failed to get container status \"b9556aacdbfe42951b1cd4063c4125571b27916ee146e1f94f981de794adca2b\": rpc error: code = NotFound desc = could not find container \"b9556aacdbfe42951b1cd4063c4125571b27916ee146e1f94f981de794adca2b\": container with ID starting with b9556aacdbfe42951b1cd4063c4125571b27916ee146e1f94f981de794adca2b not found: ID does not exist"
Sep 16 18:05:41.007094 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:41.007056    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz"]
Sep 16 18:05:41.007260 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:41.007112    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6d65c6948c-fckdz"]
Sep 16 18:05:41.082394 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:41.082360    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="955d31df-891d-49f3-bb85-728f61bcfb5a" path="/var/lib/kubelet/pods/955d31df-891d-49f3-bb85-728f61bcfb5a/volumes"
Sep 16 18:05:41.082815 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:05:41.082802    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2d0f636-c265-40a2-be7d-3085e2e62564" path="/var/lib/kubelet/pods/f2d0f636-c265-40a2-be7d-3085e2e62564/volumes"
Sep 16 18:06:03.287431 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:03.287386    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:06:03.424875 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:03.424823    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:06:03.527522 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:03.527487    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:06:03.758978 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:03.758921    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:06:04.950071 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:04.950027    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:06:05.682076 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:05.682040    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:06:08.367405 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:08.367372    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:06:26.204138 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:26.204093    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:06:26.487485 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:26.487402    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:06:26.582940 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:26.582894    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:06:33.862229 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:33.862129    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:06:40.645242 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:40.645192    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:06:40.754084 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:40.754046    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:06:46.864425 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:46.864392    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:06:49.371528 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.371495    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"]
Sep 16 18:06:49.371925 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.371898    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2d0f636-c265-40a2-be7d-3085e2e62564" containerName="storage-initializer"
Sep 16 18:06:49.371925 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.371913    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2d0f636-c265-40a2-be7d-3085e2e62564" containerName="storage-initializer"
Sep 16 18:06:49.372015 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.371930    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="955d31df-891d-49f3-bb85-728f61bcfb5a" containerName="storage-initializer"
Sep 16 18:06:49.372015 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.371935    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="955d31df-891d-49f3-bb85-728f61bcfb5a" containerName="storage-initializer"
Sep 16 18:06:49.372015 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.371942    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="955d31df-891d-49f3-bb85-728f61bcfb5a" containerName="main"
Sep 16 18:06:49.372015 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.371948    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="955d31df-891d-49f3-bb85-728f61bcfb5a" containerName="main"
Sep 16 18:06:49.372015 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.371959    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2d0f636-c265-40a2-be7d-3085e2e62564" containerName="main"
Sep 16 18:06:49.372015 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.371964    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2d0f636-c265-40a2-be7d-3085e2e62564" containerName="main"
Sep 16 18:06:49.372194 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.372048    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="955d31df-891d-49f3-bb85-728f61bcfb5a" containerName="main"
Sep 16 18:06:49.372194 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.372058    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2d0f636-c265-40a2-be7d-3085e2e62564" containerName="main"
Sep 16 18:06:49.379061 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.377510    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"
Sep 16 18:06:49.381485 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.381462    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 16 18:06:49.381773 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.381757    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 16 18:06:49.381893 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.381805    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 16 18:06:49.382906 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.382887    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:06:49.383021 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.382928    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 16 18:06:49.386300 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.386280    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-q6qv9\""
Sep 16 18:06:49.397450 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.397427    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"]
Sep 16 18:06:49.425280 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.425225    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:06:49.517453 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.517415    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54f73eb-86f8-48fc-a4db-048b37570c6c-tls-certs\") pod \"tokenizer-template-args-test-kserve-5968d8df69-t5dcp\" (UID: \"e54f73eb-86f8-48fc-a4db-048b37570c6c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"
Sep 16 18:06:49.517453 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.517454    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54f73eb-86f8-48fc-a4db-048b37570c6c-model-cache\") pod \"tokenizer-template-args-test-kserve-5968d8df69-t5dcp\" (UID: \"e54f73eb-86f8-48fc-a4db-048b37570c6c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"
Sep 16 18:06:49.517676 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.517489    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e54f73eb-86f8-48fc-a4db-048b37570c6c-home\") pod \"tokenizer-template-args-test-kserve-5968d8df69-t5dcp\" (UID: \"e54f73eb-86f8-48fc-a4db-048b37570c6c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"
Sep 16 18:06:49.517676 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.517540    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54f73eb-86f8-48fc-a4db-048b37570c6c-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5968d8df69-t5dcp\" (UID: \"e54f73eb-86f8-48fc-a4db-048b37570c6c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"
Sep 16 18:06:49.517676 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.517597    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e54f73eb-86f8-48fc-a4db-048b37570c6c-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5968d8df69-t5dcp\" (UID: \"e54f73eb-86f8-48fc-a4db-048b37570c6c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"
Sep 16 18:06:49.517676 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.517647    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e54f73eb-86f8-48fc-a4db-048b37570c6c-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5968d8df69-t5dcp\" (UID: \"e54f73eb-86f8-48fc-a4db-048b37570c6c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"
Sep 16 18:06:49.517929 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.517749    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54f73eb-86f8-48fc-a4db-048b37570c6c-dshm\") pod \"tokenizer-template-args-test-kserve-5968d8df69-t5dcp\" (UID: \"e54f73eb-86f8-48fc-a4db-048b37570c6c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"
Sep 16 18:06:49.517929 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.517797    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5vl7\" (UniqueName: \"kubernetes.io/projected/e54f73eb-86f8-48fc-a4db-048b37570c6c-kube-api-access-g5vl7\") pod \"tokenizer-template-args-test-kserve-5968d8df69-t5dcp\" (UID: \"e54f73eb-86f8-48fc-a4db-048b37570c6c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"
Sep 16 18:06:49.618433 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.618393    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e54f73eb-86f8-48fc-a4db-048b37570c6c-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5968d8df69-t5dcp\" (UID: \"e54f73eb-86f8-48fc-a4db-048b37570c6c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"
Sep 16 18:06:49.618632 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.618447    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54f73eb-86f8-48fc-a4db-048b37570c6c-dshm\") pod \"tokenizer-template-args-test-kserve-5968d8df69-t5dcp\" (UID: \"e54f73eb-86f8-48fc-a4db-048b37570c6c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"
Sep 16 18:06:49.618632 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.618471    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5vl7\" (UniqueName: \"kubernetes.io/projected/e54f73eb-86f8-48fc-a4db-048b37570c6c-kube-api-access-g5vl7\") pod \"tokenizer-template-args-test-kserve-5968d8df69-t5dcp\" (UID: \"e54f73eb-86f8-48fc-a4db-048b37570c6c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"
Sep 16 18:06:49.618632 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.618501    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54f73eb-86f8-48fc-a4db-048b37570c6c-tls-certs\") pod \"tokenizer-template-args-test-kserve-5968d8df69-t5dcp\" (UID: \"e54f73eb-86f8-48fc-a4db-048b37570c6c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"
Sep 16 18:06:49.618632 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.618520    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54f73eb-86f8-48fc-a4db-048b37570c6c-model-cache\") pod \"tokenizer-template-args-test-kserve-5968d8df69-t5dcp\" (UID: \"e54f73eb-86f8-48fc-a4db-048b37570c6c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"
Sep 16 18:06:49.618632 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.618543    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e54f73eb-86f8-48fc-a4db-048b37570c6c-home\") pod \"tokenizer-template-args-test-kserve-5968d8df69-t5dcp\" (UID: \"e54f73eb-86f8-48fc-a4db-048b37570c6c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"
Sep 16 18:06:49.618632 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.618563    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54f73eb-86f8-48fc-a4db-048b37570c6c-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5968d8df69-t5dcp\" (UID: \"e54f73eb-86f8-48fc-a4db-048b37570c6c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"
Sep 16 18:06:49.618632 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.618591    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e54f73eb-86f8-48fc-a4db-048b37570c6c-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5968d8df69-t5dcp\" (UID: \"e54f73eb-86f8-48fc-a4db-048b37570c6c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"
Sep 16 18:06:49.619005 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.618980    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54f73eb-86f8-48fc-a4db-048b37570c6c-model-cache\") pod \"tokenizer-template-args-test-kserve-5968d8df69-t5dcp\" (UID: \"e54f73eb-86f8-48fc-a4db-048b37570c6c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"
Sep 16 18:06:49.619077 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.619035    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e54f73eb-86f8-48fc-a4db-048b37570c6c-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5968d8df69-t5dcp\" (UID: \"e54f73eb-86f8-48fc-a4db-048b37570c6c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"
Sep 16 18:06:49.619232 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.619199    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e54f73eb-86f8-48fc-a4db-048b37570c6c-home\") pod \"tokenizer-template-args-test-kserve-5968d8df69-t5dcp\" (UID: \"e54f73eb-86f8-48fc-a4db-048b37570c6c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"
Sep 16 18:06:49.619232 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.619220    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54f73eb-86f8-48fc-a4db-048b37570c6c-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5968d8df69-t5dcp\" (UID: \"e54f73eb-86f8-48fc-a4db-048b37570c6c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"
Sep 16 18:06:49.619465 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.619443    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e54f73eb-86f8-48fc-a4db-048b37570c6c-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5968d8df69-t5dcp\" (UID: \"e54f73eb-86f8-48fc-a4db-048b37570c6c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"
Sep 16 18:06:49.620996 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.620968    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54f73eb-86f8-48fc-a4db-048b37570c6c-dshm\") pod \"tokenizer-template-args-test-kserve-5968d8df69-t5dcp\" (UID: \"e54f73eb-86f8-48fc-a4db-048b37570c6c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"
Sep 16 18:06:49.621282 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.621259    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54f73eb-86f8-48fc-a4db-048b37570c6c-tls-certs\") pod \"tokenizer-template-args-test-kserve-5968d8df69-t5dcp\" (UID: \"e54f73eb-86f8-48fc-a4db-048b37570c6c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"
Sep 16 18:06:49.636736 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.636677    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5vl7\" (UniqueName: \"kubernetes.io/projected/e54f73eb-86f8-48fc-a4db-048b37570c6c-kube-api-access-g5vl7\") pod \"tokenizer-template-args-test-kserve-5968d8df69-t5dcp\" (UID: \"e54f73eb-86f8-48fc-a4db-048b37570c6c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"
Sep 16 18:06:49.689246 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.689209    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"
Sep 16 18:06:49.768969 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.768939    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw"]
Sep 16 18:06:49.773420 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.773398    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw"
Sep 16 18:06:49.785708 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.785678    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw"]
Sep 16 18:06:49.822316 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.822286    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:06:49.921164 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.921069    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cd5a123-ffbb-47f4-9cc5-40180edee906-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw\" (UID: \"5cd5a123-ffbb-47f4-9cc5-40180edee906\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw"
Sep 16 18:06:49.921164 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.921112    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cd5a123-ffbb-47f4-9cc5-40180edee906-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw\" (UID: \"5cd5a123-ffbb-47f4-9cc5-40180edee906\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw"
Sep 16 18:06:49.921164 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.921146    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cd5a123-ffbb-47f4-9cc5-40180edee906-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw\" (UID: \"5cd5a123-ffbb-47f4-9cc5-40180edee906\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw"
Sep 16 18:06:49.921394 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.921265    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cd5a123-ffbb-47f4-9cc5-40180edee906-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw\" (UID: \"5cd5a123-ffbb-47f4-9cc5-40180edee906\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw"
Sep 16 18:06:49.980009 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:49.979979    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"]
Sep 16 18:06:49.981905 ip-10-0-136-122 kubenswrapper[2577]: W0916 18:06:49.981842    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode54f73eb_86f8_48fc_a4db_048b37570c6c.slice/crio-4cb729d7fd55e96e96cfc06ec5e34ada7cd0c9e6fbd859b7e6853080cc76ddf4 WatchSource:0}: Error finding container 4cb729d7fd55e96e96cfc06ec5e34ada7cd0c9e6fbd859b7e6853080cc76ddf4: Status 404 returned error can't find the container with id 4cb729d7fd55e96e96cfc06ec5e34ada7cd0c9e6fbd859b7e6853080cc76ddf4
Sep 16 18:06:50.022093 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:50.022061    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cd5a123-ffbb-47f4-9cc5-40180edee906-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw\" (UID: \"5cd5a123-ffbb-47f4-9cc5-40180edee906\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw"
Sep 16 18:06:50.022199 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:50.022120    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cd5a123-ffbb-47f4-9cc5-40180edee906-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw\" (UID: \"5cd5a123-ffbb-47f4-9cc5-40180edee906\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw"
Sep 16 18:06:50.022240 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:50.022218    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cd5a123-ffbb-47f4-9cc5-40180edee906-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw\" (UID: \"5cd5a123-ffbb-47f4-9cc5-40180edee906\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw"
Sep 16 18:06:50.022305 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:50.022291    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cd5a123-ffbb-47f4-9cc5-40180edee906-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw\" (UID: \"5cd5a123-ffbb-47f4-9cc5-40180edee906\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw"
Sep 16 18:06:50.022486 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:50.022462    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cd5a123-ffbb-47f4-9cc5-40180edee906-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw\" (UID: \"5cd5a123-ffbb-47f4-9cc5-40180edee906\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw"
Sep 16 18:06:50.022639 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:50.022614    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cd5a123-ffbb-47f4-9cc5-40180edee906-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw\" (UID: \"5cd5a123-ffbb-47f4-9cc5-40180edee906\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw"
Sep 16 18:06:50.022896 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:50.022876    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cd5a123-ffbb-47f4-9cc5-40180edee906-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw\" (UID: \"5cd5a123-ffbb-47f4-9cc5-40180edee906\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw"
Sep 16 18:06:50.024778 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:50.024750    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cd5a123-ffbb-47f4-9cc5-40180edee906-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw\" (UID: \"5cd5a123-ffbb-47f4-9cc5-40180edee906\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw"
Sep 16 18:06:50.085616 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:50.085580    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw"
Sep 16 18:06:50.168550 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:50.168515    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:06:50.340792 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:50.340755    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp" event={"ID":"e54f73eb-86f8-48fc-a4db-048b37570c6c","Type":"ContainerStarted","Data":"c74ec6d187eaee17f9d3184a904d423f0536f7c6954e72b3d4842bcb71340b49"}
Sep 16 18:06:50.341005 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:50.340818    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp" event={"ID":"e54f73eb-86f8-48fc-a4db-048b37570c6c","Type":"ContainerStarted","Data":"4cb729d7fd55e96e96cfc06ec5e34ada7cd0c9e6fbd859b7e6853080cc76ddf4"}
Sep 16 18:06:50.379773 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:50.379738    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw"]
Sep 16 18:06:50.380541 ip-10-0-136-122 kubenswrapper[2577]: W0916 18:06:50.380514    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd5a123_ffbb_47f4_9cc5_40180edee906.slice/crio-3fa5e3a9f44a4cbe29c576fa81dc1d53201684b2a9807d70d5e0301e04a3b394 WatchSource:0}: Error finding container 3fa5e3a9f44a4cbe29c576fa81dc1d53201684b2a9807d70d5e0301e04a3b394: Status 404 returned error can't find the container with id 3fa5e3a9f44a4cbe29c576fa81dc1d53201684b2a9807d70d5e0301e04a3b394
Sep 16 18:06:51.349989 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:51.349949    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw" event={"ID":"5cd5a123-ffbb-47f4-9cc5-40180edee906","Type":"ContainerStarted","Data":"05c23dcdd6254f6afb4aea10368ae7d3034132bc29ff55c91d11fffe3457915a"}
Sep 16 18:06:51.350175 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:51.350012    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw" event={"ID":"5cd5a123-ffbb-47f4-9cc5-40180edee906","Type":"ContainerStarted","Data":"3fa5e3a9f44a4cbe29c576fa81dc1d53201684b2a9807d70d5e0301e04a3b394"}
Sep 16 18:06:53.362957 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:53.362918    2577 generic.go:356] "Generic (PLEG): container finished" podID="5cd5a123-ffbb-47f4-9cc5-40180edee906" containerID="05c23dcdd6254f6afb4aea10368ae7d3034132bc29ff55c91d11fffe3457915a" exitCode=0
Sep 16 18:06:53.363408 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:53.362993    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw" event={"ID":"5cd5a123-ffbb-47f4-9cc5-40180edee906","Type":"ContainerDied","Data":"05c23dcdd6254f6afb4aea10368ae7d3034132bc29ff55c91d11fffe3457915a"}
Sep 16 18:06:53.365095 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:53.365074    2577 generic.go:356] "Generic (PLEG): container finished" podID="e54f73eb-86f8-48fc-a4db-048b37570c6c" containerID="c74ec6d187eaee17f9d3184a904d423f0536f7c6954e72b3d4842bcb71340b49" exitCode=0
Sep 16 18:06:53.365223 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:53.365105    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp" event={"ID":"e54f73eb-86f8-48fc-a4db-048b37570c6c","Type":"ContainerDied","Data":"c74ec6d187eaee17f9d3184a904d423f0536f7c6954e72b3d4842bcb71340b49"}
Sep 16 18:06:53.970667 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:53.970628    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:06:54.082550 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:54.082514    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:06:54.373096 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:54.373065    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp" event={"ID":"e54f73eb-86f8-48fc-a4db-048b37570c6c","Type":"ContainerStarted","Data":"2c739150562acae24425f7c4addc16a71dab2130d16cd96833085d84f3cbd41e"}
Sep 16 18:06:54.374938 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:54.374902    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw" event={"ID":"5cd5a123-ffbb-47f4-9cc5-40180edee906","Type":"ContainerStarted","Data":"979d10049d122180cb79fe8bc72a439c3f97aa98017de2809327f5b2a0a28c03"}
Sep 16 18:06:54.375120 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:54.375100    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw"
Sep 16 18:06:54.412181 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:54.412119    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp" podStartSLOduration=2.844104427 podStartE2EDuration="5.412105731s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.568001304s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:06:54.406287483 +0000 UTC m=+2535.768966280" watchObservedRunningTime="2026-09-16 18:06:54.412105731 +0000 UTC m=+2535.774784516"
Sep 16 18:06:54.444036 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:54.443979    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw" podStartSLOduration=2.935553371 podStartE2EDuration="5.443962022s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.508408651s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:06:54.441458719 +0000 UTC m=+2535.804137506" watchObservedRunningTime="2026-09-16 18:06:54.443962022 +0000 UTC m=+2535.806640808"
Sep 16 18:06:54.459772 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:54.459737    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:06:54.689747 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:54.689656    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"
Sep 16 18:06:58.544661 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:58.544623    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:06:59.689799 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:59.689761    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"
Sep 16 18:06:59.708725 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:06:59.708695    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"
Sep 16 18:07:00.420640 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:07:00.420611    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"
Sep 16 18:07:15.677026 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:07:15.676989    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:07:25.392509 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:07:25.392466    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw"
Sep 16 18:08:02.363751 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:02.363661    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:08:04.483365 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:04.483325    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw_5cd5a123-ffbb-47f4-9cc5-40180edee906/main/0.log"
Sep 16 18:08:04.736296 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:04.736202    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:08:07.094172 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:08:07.094135    2577 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-self-signed-certs: secret "tokenizer-template-args-test-kserve-self-signed-certs" not found
Sep 16 18:08:07.094574 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:08:07.094205    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e54f73eb-86f8-48fc-a4db-048b37570c6c-tls-certs podName:e54f73eb-86f8-48fc-a4db-048b37570c6c nodeName:}" failed. No retries permitted until 2026-09-16 18:08:07.59418846 +0000 UTC m=+2608.956867224 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e54f73eb-86f8-48fc-a4db-048b37570c6c-tls-certs") pod "tokenizer-template-args-test-kserve-5968d8df69-t5dcp" (UID: "e54f73eb-86f8-48fc-a4db-048b37570c6c") : secret "tokenizer-template-args-test-kserve-self-signed-certs" not found
Sep 16 18:08:07.129454 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:07.129410    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:08:07.130182 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:07.130156    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw"]
Sep 16 18:08:07.130422 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:07.130400    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw" podUID="5cd5a123-ffbb-47f4-9cc5-40180edee906" containerName="main" containerID="cri-o://979d10049d122180cb79fe8bc72a439c3f97aa98017de2809327f5b2a0a28c03" gracePeriod=30
Sep 16 18:08:07.135370 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:07.135346    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"]
Sep 16 18:08:07.226977 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:07.226937    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:08:07.597557 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:08:07.597516    2577 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-self-signed-certs: secret "tokenizer-template-args-test-kserve-self-signed-certs" not found
Sep 16 18:08:07.597741 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:08:07.597605    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e54f73eb-86f8-48fc-a4db-048b37570c6c-tls-certs podName:e54f73eb-86f8-48fc-a4db-048b37570c6c nodeName:}" failed. No retries permitted until 2026-09-16 18:08:08.597584331 +0000 UTC m=+2609.960263099 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e54f73eb-86f8-48fc-a4db-048b37570c6c-tls-certs") pod "tokenizer-template-args-test-kserve-5968d8df69-t5dcp" (UID: "e54f73eb-86f8-48fc-a4db-048b37570c6c") : secret "tokenizer-template-args-test-kserve-self-signed-certs" not found
Sep 16 18:08:07.778663 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:07.778624    2577 generic.go:356] "Generic (PLEG): container finished" podID="5cd5a123-ffbb-47f4-9cc5-40180edee906" containerID="979d10049d122180cb79fe8bc72a439c3f97aa98017de2809327f5b2a0a28c03" exitCode=0
Sep 16 18:08:07.778807 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:07.778682    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw" event={"ID":"5cd5a123-ffbb-47f4-9cc5-40180edee906","Type":"ContainerDied","Data":"979d10049d122180cb79fe8bc72a439c3f97aa98017de2809327f5b2a0a28c03"}
Sep 16 18:08:07.778807 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:07.778738    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw" event={"ID":"5cd5a123-ffbb-47f4-9cc5-40180edee906","Type":"ContainerDied","Data":"3fa5e3a9f44a4cbe29c576fa81dc1d53201684b2a9807d70d5e0301e04a3b394"}
Sep 16 18:08:07.778807 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:07.778756    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fa5e3a9f44a4cbe29c576fa81dc1d53201684b2a9807d70d5e0301e04a3b394"
Sep 16 18:08:07.786508 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:07.786489    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw"
Sep 16 18:08:07.787629 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:08:07.787606    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 18:08:07.787629 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:08:07.787629 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:08:07.787629 ip-10-0-136-122 kubenswrapper[2577]:  > containerID="2c739150562acae24425f7c4addc16a71dab2130d16cd96833085d84f3cbd41e" cmd=["/bin/sleep","15"]
Sep 16 18:08:07.787783 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:07.787650    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp" message="" err=<
Sep 16 18:08:07.787783 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:08:07.787783 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:08:07.787783 ip-10-0-136-122 kubenswrapper[2577]:  >
Sep 16 18:08:07.787783 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:08:07.787662    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 18:08:07.787783 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:08:07.787783 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:08:07.787783 ip-10-0-136-122 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp" podUID="e54f73eb-86f8-48fc-a4db-048b37570c6c" containerName="main" containerID="cri-o://2c739150562acae24425f7c4addc16a71dab2130d16cd96833085d84f3cbd41e"
Sep 16 18:08:07.787783 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:07.787694    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp" podUID="e54f73eb-86f8-48fc-a4db-048b37570c6c" containerName="main" containerID="cri-o://2c739150562acae24425f7c4addc16a71dab2130d16cd96833085d84f3cbd41e" gracePeriod=60
Sep 16 18:08:07.798453 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:07.798432    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cd5a123-ffbb-47f4-9cc5-40180edee906-kserve-provision-location\") pod \"5cd5a123-ffbb-47f4-9cc5-40180edee906\" (UID: \"5cd5a123-ffbb-47f4-9cc5-40180edee906\") "
Sep 16 18:08:07.798573 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:07.798490    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cd5a123-ffbb-47f4-9cc5-40180edee906-tls-certs\") pod \"5cd5a123-ffbb-47f4-9cc5-40180edee906\" (UID: \"5cd5a123-ffbb-47f4-9cc5-40180edee906\") "
Sep 16 18:08:07.798573 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:07.798516    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cd5a123-ffbb-47f4-9cc5-40180edee906-cabundle-cert\") pod \"5cd5a123-ffbb-47f4-9cc5-40180edee906\" (UID: \"5cd5a123-ffbb-47f4-9cc5-40180edee906\") "
Sep 16 18:08:07.798573 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:07.798567    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cd5a123-ffbb-47f4-9cc5-40180edee906-tmp-dir\") pod \"5cd5a123-ffbb-47f4-9cc5-40180edee906\" (UID: \"5cd5a123-ffbb-47f4-9cc5-40180edee906\") "
Sep 16 18:08:07.798963 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:07.798937    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cd5a123-ffbb-47f4-9cc5-40180edee906-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5cd5a123-ffbb-47f4-9cc5-40180edee906" (UID: "5cd5a123-ffbb-47f4-9cc5-40180edee906"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:08:07.799213 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:07.799191    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd5a123-ffbb-47f4-9cc5-40180edee906-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5cd5a123-ffbb-47f4-9cc5-40180edee906" (UID: "5cd5a123-ffbb-47f4-9cc5-40180edee906"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:08:07.800673 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:07.800649    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cd5a123-ffbb-47f4-9cc5-40180edee906-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5cd5a123-ffbb-47f4-9cc5-40180edee906" (UID: "5cd5a123-ffbb-47f4-9cc5-40180edee906"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:08:07.825311 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:07.825277    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd5a123-ffbb-47f4-9cc5-40180edee906-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5cd5a123-ffbb-47f4-9cc5-40180edee906" (UID: "5cd5a123-ffbb-47f4-9cc5-40180edee906"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:08:07.899546 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:07.899512    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cd5a123-ffbb-47f4-9cc5-40180edee906-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:08:07.899546 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:07.899543    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cd5a123-ffbb-47f4-9cc5-40180edee906-cabundle-cert\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:08:07.899701 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:07.899556    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cd5a123-ffbb-47f4-9cc5-40180edee906-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:08:07.899701 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:07.899568    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cd5a123-ffbb-47f4-9cc5-40180edee906-kserve-provision-location\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:08:08.128789 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.128644    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"
Sep 16 18:08:08.201577 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.201542    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5vl7\" (UniqueName: \"kubernetes.io/projected/e54f73eb-86f8-48fc-a4db-048b37570c6c-kube-api-access-g5vl7\") pod \"e54f73eb-86f8-48fc-a4db-048b37570c6c\" (UID: \"e54f73eb-86f8-48fc-a4db-048b37570c6c\") "
Sep 16 18:08:08.201762 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.201590    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54f73eb-86f8-48fc-a4db-048b37570c6c-model-cache\") pod \"e54f73eb-86f8-48fc-a4db-048b37570c6c\" (UID: \"e54f73eb-86f8-48fc-a4db-048b37570c6c\") "
Sep 16 18:08:08.201762 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.201626    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54f73eb-86f8-48fc-a4db-048b37570c6c-tmp-dir\") pod \"e54f73eb-86f8-48fc-a4db-048b37570c6c\" (UID: \"e54f73eb-86f8-48fc-a4db-048b37570c6c\") "
Sep 16 18:08:08.201762 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.201641    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e54f73eb-86f8-48fc-a4db-048b37570c6c-home\") pod \"e54f73eb-86f8-48fc-a4db-048b37570c6c\" (UID: \"e54f73eb-86f8-48fc-a4db-048b37570c6c\") "
Sep 16 18:08:08.201762 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.201659    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e54f73eb-86f8-48fc-a4db-048b37570c6c-kserve-provision-location\") pod \"e54f73eb-86f8-48fc-a4db-048b37570c6c\" (UID: \"e54f73eb-86f8-48fc-a4db-048b37570c6c\") "
Sep 16 18:08:08.201762 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.201705    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e54f73eb-86f8-48fc-a4db-048b37570c6c-cabundle-cert\") pod \"e54f73eb-86f8-48fc-a4db-048b37570c6c\" (UID: \"e54f73eb-86f8-48fc-a4db-048b37570c6c\") "
Sep 16 18:08:08.201762 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.201743    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54f73eb-86f8-48fc-a4db-048b37570c6c-dshm\") pod \"e54f73eb-86f8-48fc-a4db-048b37570c6c\" (UID: \"e54f73eb-86f8-48fc-a4db-048b37570c6c\") "
Sep 16 18:08:08.202109 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.201787    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54f73eb-86f8-48fc-a4db-048b37570c6c-tls-certs\") pod \"e54f73eb-86f8-48fc-a4db-048b37570c6c\" (UID: \"e54f73eb-86f8-48fc-a4db-048b37570c6c\") "
Sep 16 18:08:08.202109 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.202007    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54f73eb-86f8-48fc-a4db-048b37570c6c-model-cache" (OuterVolumeSpecName: "model-cache") pod "e54f73eb-86f8-48fc-a4db-048b37570c6c" (UID: "e54f73eb-86f8-48fc-a4db-048b37570c6c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:08:08.202109 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.202017    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54f73eb-86f8-48fc-a4db-048b37570c6c-home" (OuterVolumeSpecName: "home") pod "e54f73eb-86f8-48fc-a4db-048b37570c6c" (UID: "e54f73eb-86f8-48fc-a4db-048b37570c6c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:08:08.202109 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.202032    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54f73eb-86f8-48fc-a4db-048b37570c6c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e54f73eb-86f8-48fc-a4db-048b37570c6c" (UID: "e54f73eb-86f8-48fc-a4db-048b37570c6c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:08:08.202238 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.202120    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e54f73eb-86f8-48fc-a4db-048b37570c6c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e54f73eb-86f8-48fc-a4db-048b37570c6c" (UID: "e54f73eb-86f8-48fc-a4db-048b37570c6c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:08:08.204006 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.203881    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54f73eb-86f8-48fc-a4db-048b37570c6c-dshm" (OuterVolumeSpecName: "dshm") pod "e54f73eb-86f8-48fc-a4db-048b37570c6c" (UID: "e54f73eb-86f8-48fc-a4db-048b37570c6c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:08:08.204132 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.204045    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e54f73eb-86f8-48fc-a4db-048b37570c6c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e54f73eb-86f8-48fc-a4db-048b37570c6c" (UID: "e54f73eb-86f8-48fc-a4db-048b37570c6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:08:08.204132 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.204080    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e54f73eb-86f8-48fc-a4db-048b37570c6c-kube-api-access-g5vl7" (OuterVolumeSpecName: "kube-api-access-g5vl7") pod "e54f73eb-86f8-48fc-a4db-048b37570c6c" (UID: "e54f73eb-86f8-48fc-a4db-048b37570c6c"). InnerVolumeSpecName "kube-api-access-g5vl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:08:08.223788 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.223762    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54f73eb-86f8-48fc-a4db-048b37570c6c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e54f73eb-86f8-48fc-a4db-048b37570c6c" (UID: "e54f73eb-86f8-48fc-a4db-048b37570c6c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:08:08.302793 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.302759    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e54f73eb-86f8-48fc-a4db-048b37570c6c-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:08:08.302793 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.302787    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e54f73eb-86f8-48fc-a4db-048b37570c6c-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:08:08.302793 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.302796    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e54f73eb-86f8-48fc-a4db-048b37570c6c-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:08:08.303029 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.302805    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e54f73eb-86f8-48fc-a4db-048b37570c6c-kserve-provision-location\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:08:08.303029 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.302815    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e54f73eb-86f8-48fc-a4db-048b37570c6c-cabundle-cert\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:08:08.303029 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.302825    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e54f73eb-86f8-48fc-a4db-048b37570c6c-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:08:08.303029 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.302833    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54f73eb-86f8-48fc-a4db-048b37570c6c-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:08:08.303029 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.302841    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5vl7\" (UniqueName: \"kubernetes.io/projected/e54f73eb-86f8-48fc-a4db-048b37570c6c-kube-api-access-g5vl7\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:08:08.786739 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.786704    2577 generic.go:356] "Generic (PLEG): container finished" podID="e54f73eb-86f8-48fc-a4db-048b37570c6c" containerID="2c739150562acae24425f7c4addc16a71dab2130d16cd96833085d84f3cbd41e" exitCode=0
Sep 16 18:08:08.786939 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.786769    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"
Sep 16 18:08:08.786939 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.786793    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp" event={"ID":"e54f73eb-86f8-48fc-a4db-048b37570c6c","Type":"ContainerDied","Data":"2c739150562acae24425f7c4addc16a71dab2130d16cd96833085d84f3cbd41e"}
Sep 16 18:08:08.786939 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.786903    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp" event={"ID":"e54f73eb-86f8-48fc-a4db-048b37570c6c","Type":"ContainerDied","Data":"4cb729d7fd55e96e96cfc06ec5e34ada7cd0c9e6fbd859b7e6853080cc76ddf4"}
Sep 16 18:08:08.786939 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.786931    2577 scope.go:117] "RemoveContainer" containerID="2c739150562acae24425f7c4addc16a71dab2130d16cd96833085d84f3cbd41e"
Sep 16 18:08:08.787161 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.787101    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw"
Sep 16 18:08:08.797907 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.797884    2577 scope.go:117] "RemoveContainer" containerID="c74ec6d187eaee17f9d3184a904d423f0536f7c6954e72b3d4842bcb71340b49"
Sep 16 18:08:08.806186 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.806167    2577 scope.go:117] "RemoveContainer" containerID="2c739150562acae24425f7c4addc16a71dab2130d16cd96833085d84f3cbd41e"
Sep 16 18:08:08.806489 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:08:08.806464    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c739150562acae24425f7c4addc16a71dab2130d16cd96833085d84f3cbd41e\": container with ID starting with 2c739150562acae24425f7c4addc16a71dab2130d16cd96833085d84f3cbd41e not found: ID does not exist" containerID="2c739150562acae24425f7c4addc16a71dab2130d16cd96833085d84f3cbd41e"
Sep 16 18:08:08.806532 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.806498    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c739150562acae24425f7c4addc16a71dab2130d16cd96833085d84f3cbd41e"} err="failed to get container status \"2c739150562acae24425f7c4addc16a71dab2130d16cd96833085d84f3cbd41e\": rpc error: code = NotFound desc = could not find container \"2c739150562acae24425f7c4addc16a71dab2130d16cd96833085d84f3cbd41e\": container with ID starting with 2c739150562acae24425f7c4addc16a71dab2130d16cd96833085d84f3cbd41e not found: ID does not exist"
Sep 16 18:08:08.806532 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.806514    2577 scope.go:117] "RemoveContainer" containerID="c74ec6d187eaee17f9d3184a904d423f0536f7c6954e72b3d4842bcb71340b49"
Sep 16 18:08:08.806763 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:08:08.806741    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c74ec6d187eaee17f9d3184a904d423f0536f7c6954e72b3d4842bcb71340b49\": container with ID starting with c74ec6d187eaee17f9d3184a904d423f0536f7c6954e72b3d4842bcb71340b49 not found: ID does not exist" containerID="c74ec6d187eaee17f9d3184a904d423f0536f7c6954e72b3d4842bcb71340b49"
Sep 16 18:08:08.806805 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.806772    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c74ec6d187eaee17f9d3184a904d423f0536f7c6954e72b3d4842bcb71340b49"} err="failed to get container status \"c74ec6d187eaee17f9d3184a904d423f0536f7c6954e72b3d4842bcb71340b49\": rpc error: code = NotFound desc = could not find container \"c74ec6d187eaee17f9d3184a904d423f0536f7c6954e72b3d4842bcb71340b49\": container with ID starting with c74ec6d187eaee17f9d3184a904d423f0536f7c6954e72b3d4842bcb71340b49 not found: ID does not exist"
Sep 16 18:08:08.826016 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.825958    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"]
Sep 16 18:08:08.838146 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.838120    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-t5dcp"]
Sep 16 18:08:08.859246 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.859220    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw"]
Sep 16 18:08:08.866848 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:08.866824    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-dc6db48c5-hv7qw"]
Sep 16 18:08:09.081701 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:09.081612    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cd5a123-ffbb-47f4-9cc5-40180edee906" path="/var/lib/kubelet/pods/5cd5a123-ffbb-47f4-9cc5-40180edee906/volumes"
Sep 16 18:08:09.082222 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:09.082200    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e54f73eb-86f8-48fc-a4db-048b37570c6c" path="/var/lib/kubelet/pods/e54f73eb-86f8-48fc-a4db-048b37570c6c/volumes"
Sep 16 18:08:34.336317 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:34.336280    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:08:34.482020 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:34.481982    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:08:34.874719 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:34.874685    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:08:35.617784 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:35.617751    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:08:39.365035 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:39.364985    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:08:54.979885 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:54.979823    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:08:56.480434 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:08:56.480401    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:09:02.035199 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:02.035162    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:09:03.023492 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:03.023458    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:09:09.230286 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:09.230246    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:09:09.328138 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:09.328101    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:09:10.032369 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:10.032333    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:09:16.880450 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:16.880411    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:09:23.345628 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:23.345593    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:09:23.470669 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:23.470633    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:09:37.174487 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.174406    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts"]
Sep 16 18:09:37.174907 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.174773    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e54f73eb-86f8-48fc-a4db-048b37570c6c" containerName="main"
Sep 16 18:09:37.174907 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.174787    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e54f73eb-86f8-48fc-a4db-048b37570c6c" containerName="main"
Sep 16 18:09:37.174907 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.174800    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e54f73eb-86f8-48fc-a4db-048b37570c6c" containerName="storage-initializer"
Sep 16 18:09:37.174907 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.174806    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e54f73eb-86f8-48fc-a4db-048b37570c6c" containerName="storage-initializer"
Sep 16 18:09:37.174907 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.174814    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cd5a123-ffbb-47f4-9cc5-40180edee906" containerName="main"
Sep 16 18:09:37.174907 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.174821    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cd5a123-ffbb-47f4-9cc5-40180edee906" containerName="main"
Sep 16 18:09:37.174907 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.174842    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cd5a123-ffbb-47f4-9cc5-40180edee906" containerName="storage-initializer"
Sep 16 18:09:37.174907 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.174849    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cd5a123-ffbb-47f4-9cc5-40180edee906" containerName="storage-initializer"
Sep 16 18:09:37.175203 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.174929    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e54f73eb-86f8-48fc-a4db-048b37570c6c" containerName="main"
Sep 16 18:09:37.175203 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.174939    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cd5a123-ffbb-47f4-9cc5-40180edee906" containerName="main"
Sep 16 18:09:37.177670 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.177646    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts"
Sep 16 18:09:37.182874 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.182839    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-s92xv\""
Sep 16 18:09:37.182996 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.182895    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 16 18:09:37.182996 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.182912    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 16 18:09:37.182996 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.182897    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 16 18:09:37.194770 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.194746    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts"]
Sep 16 18:09:37.226308 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.226256    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:09:37.246519 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.246493    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6a309f7-102b-479f-9d0f-deb2c007804a-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts\" (UID: \"d6a309f7-102b-479f-9d0f-deb2c007804a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts"
Sep 16 18:09:37.246646 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.246524    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9wjm\" (UniqueName: \"kubernetes.io/projected/d6a309f7-102b-479f-9d0f-deb2c007804a-kube-api-access-t9wjm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts\" (UID: \"d6a309f7-102b-479f-9d0f-deb2c007804a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts"
Sep 16 18:09:37.246646 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.246552    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6a309f7-102b-479f-9d0f-deb2c007804a-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts\" (UID: \"d6a309f7-102b-479f-9d0f-deb2c007804a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts"
Sep 16 18:09:37.246646 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.246571    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6a309f7-102b-479f-9d0f-deb2c007804a-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts\" (UID: \"d6a309f7-102b-479f-9d0f-deb2c007804a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts"
Sep 16 18:09:37.246646 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.246632    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6a309f7-102b-479f-9d0f-deb2c007804a-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts\" (UID: \"d6a309f7-102b-479f-9d0f-deb2c007804a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts"
Sep 16 18:09:37.246800 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.246663    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6a309f7-102b-479f-9d0f-deb2c007804a-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts\" (UID: \"d6a309f7-102b-479f-9d0f-deb2c007804a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts"
Sep 16 18:09:37.347375 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.347333    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6a309f7-102b-479f-9d0f-deb2c007804a-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts\" (UID: \"d6a309f7-102b-479f-9d0f-deb2c007804a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts"
Sep 16 18:09:37.347375 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.347376    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9wjm\" (UniqueName: \"kubernetes.io/projected/d6a309f7-102b-479f-9d0f-deb2c007804a-kube-api-access-t9wjm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts\" (UID: \"d6a309f7-102b-479f-9d0f-deb2c007804a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts"
Sep 16 18:09:37.347600 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.347402    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6a309f7-102b-479f-9d0f-deb2c007804a-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts\" (UID: \"d6a309f7-102b-479f-9d0f-deb2c007804a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts"
Sep 16 18:09:37.347600 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.347421    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6a309f7-102b-479f-9d0f-deb2c007804a-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts\" (UID: \"d6a309f7-102b-479f-9d0f-deb2c007804a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts"
Sep 16 18:09:37.347600 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.347455    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6a309f7-102b-479f-9d0f-deb2c007804a-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts\" (UID: \"d6a309f7-102b-479f-9d0f-deb2c007804a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts"
Sep 16 18:09:37.347600 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.347477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6a309f7-102b-479f-9d0f-deb2c007804a-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts\" (UID: \"d6a309f7-102b-479f-9d0f-deb2c007804a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts"
Sep 16 18:09:37.347885 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.347837    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6a309f7-102b-479f-9d0f-deb2c007804a-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts\" (UID: \"d6a309f7-102b-479f-9d0f-deb2c007804a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts"
Sep 16 18:09:37.347885 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.347855    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6a309f7-102b-479f-9d0f-deb2c007804a-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts\" (UID: \"d6a309f7-102b-479f-9d0f-deb2c007804a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts"
Sep 16 18:09:37.347968 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.347921    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6a309f7-102b-479f-9d0f-deb2c007804a-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts\" (UID: \"d6a309f7-102b-479f-9d0f-deb2c007804a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts"
Sep 16 18:09:37.349754 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.349736    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6a309f7-102b-479f-9d0f-deb2c007804a-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts\" (UID: \"d6a309f7-102b-479f-9d0f-deb2c007804a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts"
Sep 16 18:09:37.350108 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.350090    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6a309f7-102b-479f-9d0f-deb2c007804a-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts\" (UID: \"d6a309f7-102b-479f-9d0f-deb2c007804a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts"
Sep 16 18:09:37.357658 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.357636    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9wjm\" (UniqueName: \"kubernetes.io/projected/d6a309f7-102b-479f-9d0f-deb2c007804a-kube-api-access-t9wjm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts\" (UID: \"d6a309f7-102b-479f-9d0f-deb2c007804a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts"
Sep 16 18:09:37.488041 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.487951    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts"
Sep 16 18:09:37.517528 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.517487    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:09:37.773797 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:37.773770    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts"]
Sep 16 18:09:37.775577 ip-10-0-136-122 kubenswrapper[2577]: W0916 18:09:37.775549    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a309f7_102b_479f_9d0f_deb2c007804a.slice/crio-3602b806ff17005a6904a5db7b0e26cab78409abdf1b4c7a2eaf8be0301b73d8 WatchSource:0}: Error finding container 3602b806ff17005a6904a5db7b0e26cab78409abdf1b4c7a2eaf8be0301b73d8: Status 404 returned error can't find the container with id 3602b806ff17005a6904a5db7b0e26cab78409abdf1b4c7a2eaf8be0301b73d8
Sep 16 18:09:38.270082 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:38.270044    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts" event={"ID":"d6a309f7-102b-479f-9d0f-deb2c007804a","Type":"ContainerStarted","Data":"98aba8b7a424f1654d4bfadea5df8903c767506e07b90afa7a3ef921f3d7419c"}
Sep 16 18:09:38.270488 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:38.270107    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts" event={"ID":"d6a309f7-102b-479f-9d0f-deb2c007804a","Type":"ContainerStarted","Data":"3602b806ff17005a6904a5db7b0e26cab78409abdf1b4c7a2eaf8be0301b73d8"}
Sep 16 18:09:38.298326 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:38.298279    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts" podStartSLOduration=1.2982633670000001 podStartE2EDuration="1.298263367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:09:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:09:38.297080519 +0000 UTC m=+2699.659759316" watchObservedRunningTime="2026-09-16 18:09:38.298263367 +0000 UTC m=+2699.660942155"
Sep 16 18:09:38.488547 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:38.488514    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts"
Sep 16 18:09:47.488910 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:47.488874    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts"
Sep 16 18:09:47.501740 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:47.501711    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts"
Sep 16 18:09:48.339269 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:09:48.339235    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts"
Sep 16 18:10:10.533812 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:10.533769    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:10:15.022153 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:15.022117    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts"]
Sep 16 18:10:15.031095 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:10:15.031041    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 18:10:15.031095 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:10:15.031095 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:10:15.031095 ip-10-0-136-122 kubenswrapper[2577]:  > containerID="98aba8b7a424f1654d4bfadea5df8903c767506e07b90afa7a3ef921f3d7419c" cmd=["/bin/sleep","15"]
Sep 16 18:10:15.031410 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:15.031120    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts" message="" err=<
Sep 16 18:10:15.031410 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:10:15.031410 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:10:15.031410 ip-10-0-136-122 kubenswrapper[2577]:  >
Sep 16 18:10:15.031410 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:10:15.031143    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 18:10:15.031410 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:10:15.031410 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:10:15.031410 ip-10-0-136-122 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts" podUID="d6a309f7-102b-479f-9d0f-deb2c007804a" containerName="main" containerID="cri-o://98aba8b7a424f1654d4bfadea5df8903c767506e07b90afa7a3ef921f3d7419c"
Sep 16 18:10:15.031410 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:15.031189    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts" podUID="d6a309f7-102b-479f-9d0f-deb2c007804a" containerName="main" containerID="cri-o://98aba8b7a424f1654d4bfadea5df8903c767506e07b90afa7a3ef921f3d7419c" gracePeriod=60
Sep 16 18:10:15.058837 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:15.058762    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:10:15.138649 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:15.138618    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:10:15.383688 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:15.383663    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts"
Sep 16 18:10:15.466409 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:15.466370    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6a309f7-102b-479f-9d0f-deb2c007804a-tmp-dir\") pod \"d6a309f7-102b-479f-9d0f-deb2c007804a\" (UID: \"d6a309f7-102b-479f-9d0f-deb2c007804a\") "
Sep 16 18:10:15.466609 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:15.466420    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9wjm\" (UniqueName: \"kubernetes.io/projected/d6a309f7-102b-479f-9d0f-deb2c007804a-kube-api-access-t9wjm\") pod \"d6a309f7-102b-479f-9d0f-deb2c007804a\" (UID: \"d6a309f7-102b-479f-9d0f-deb2c007804a\") "
Sep 16 18:10:15.466609 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:15.466447    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6a309f7-102b-479f-9d0f-deb2c007804a-dshm\") pod \"d6a309f7-102b-479f-9d0f-deb2c007804a\" (UID: \"d6a309f7-102b-479f-9d0f-deb2c007804a\") "
Sep 16 18:10:15.466609 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:15.466488    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6a309f7-102b-479f-9d0f-deb2c007804a-model-cache\") pod \"d6a309f7-102b-479f-9d0f-deb2c007804a\" (UID: \"d6a309f7-102b-479f-9d0f-deb2c007804a\") "
Sep 16 18:10:15.466609 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:15.466513    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6a309f7-102b-479f-9d0f-deb2c007804a-home\") pod \"d6a309f7-102b-479f-9d0f-deb2c007804a\" (UID: \"d6a309f7-102b-479f-9d0f-deb2c007804a\") "
Sep 16 18:10:15.466609 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:15.466567    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6a309f7-102b-479f-9d0f-deb2c007804a-tls-certs\") pod \"d6a309f7-102b-479f-9d0f-deb2c007804a\" (UID: \"d6a309f7-102b-479f-9d0f-deb2c007804a\") "
Sep 16 18:10:15.466902 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:15.466681    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a309f7-102b-479f-9d0f-deb2c007804a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d6a309f7-102b-479f-9d0f-deb2c007804a" (UID: "d6a309f7-102b-479f-9d0f-deb2c007804a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:10:15.466902 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:15.466782    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a309f7-102b-479f-9d0f-deb2c007804a-model-cache" (OuterVolumeSpecName: "model-cache") pod "d6a309f7-102b-479f-9d0f-deb2c007804a" (UID: "d6a309f7-102b-479f-9d0f-deb2c007804a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:10:15.466902 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:15.466805    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6a309f7-102b-479f-9d0f-deb2c007804a-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:10:15.466902 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:15.466886    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a309f7-102b-479f-9d0f-deb2c007804a-home" (OuterVolumeSpecName: "home") pod "d6a309f7-102b-479f-9d0f-deb2c007804a" (UID: "d6a309f7-102b-479f-9d0f-deb2c007804a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:10:15.468773 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:15.468753    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a309f7-102b-479f-9d0f-deb2c007804a-dshm" (OuterVolumeSpecName: "dshm") pod "d6a309f7-102b-479f-9d0f-deb2c007804a" (UID: "d6a309f7-102b-479f-9d0f-deb2c007804a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:10:15.468929 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:15.468909    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6a309f7-102b-479f-9d0f-deb2c007804a-kube-api-access-t9wjm" (OuterVolumeSpecName: "kube-api-access-t9wjm") pod "d6a309f7-102b-479f-9d0f-deb2c007804a" (UID: "d6a309f7-102b-479f-9d0f-deb2c007804a"). InnerVolumeSpecName "kube-api-access-t9wjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:10:15.469047 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:15.468989    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6a309f7-102b-479f-9d0f-deb2c007804a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d6a309f7-102b-479f-9d0f-deb2c007804a" (UID: "d6a309f7-102b-479f-9d0f-deb2c007804a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:10:15.474084 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:15.473930    2577 generic.go:356] "Generic (PLEG): container finished" podID="d6a309f7-102b-479f-9d0f-deb2c007804a" containerID="98aba8b7a424f1654d4bfadea5df8903c767506e07b90afa7a3ef921f3d7419c" exitCode=0
Sep 16 18:10:15.474084 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:15.473972    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts" event={"ID":"d6a309f7-102b-479f-9d0f-deb2c007804a","Type":"ContainerDied","Data":"98aba8b7a424f1654d4bfadea5df8903c767506e07b90afa7a3ef921f3d7419c"}
Sep 16 18:10:15.474084 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:15.473986    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts"
Sep 16 18:10:15.474084 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:15.474014    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts" event={"ID":"d6a309f7-102b-479f-9d0f-deb2c007804a","Type":"ContainerDied","Data":"3602b806ff17005a6904a5db7b0e26cab78409abdf1b4c7a2eaf8be0301b73d8"}
Sep 16 18:10:15.474084 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:15.474034    2577 scope.go:117] "RemoveContainer" containerID="98aba8b7a424f1654d4bfadea5df8903c767506e07b90afa7a3ef921f3d7419c"
Sep 16 18:10:15.489783 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:15.489763    2577 scope.go:117] "RemoveContainer" containerID="98aba8b7a424f1654d4bfadea5df8903c767506e07b90afa7a3ef921f3d7419c"
Sep 16 18:10:15.490098 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:10:15.490073    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98aba8b7a424f1654d4bfadea5df8903c767506e07b90afa7a3ef921f3d7419c\": container with ID starting with 98aba8b7a424f1654d4bfadea5df8903c767506e07b90afa7a3ef921f3d7419c not found: ID does not exist" containerID="98aba8b7a424f1654d4bfadea5df8903c767506e07b90afa7a3ef921f3d7419c"
Sep 16 18:10:15.490176 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:15.490104    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98aba8b7a424f1654d4bfadea5df8903c767506e07b90afa7a3ef921f3d7419c"} err="failed to get container status \"98aba8b7a424f1654d4bfadea5df8903c767506e07b90afa7a3ef921f3d7419c\": rpc error: code = NotFound desc = could not find container \"98aba8b7a424f1654d4bfadea5df8903c767506e07b90afa7a3ef921f3d7419c\": container with ID starting with 98aba8b7a424f1654d4bfadea5df8903c767506e07b90afa7a3ef921f3d7419c not found: ID does not exist"
Sep 16 18:10:15.507637 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:15.507612    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts"]
Sep 16 18:10:15.510396 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:15.510376    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts"]
Sep 16 18:10:15.567991 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:15.567963    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9wjm\" (UniqueName: \"kubernetes.io/projected/d6a309f7-102b-479f-9d0f-deb2c007804a-kube-api-access-t9wjm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:10:15.567991 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:15.567990    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6a309f7-102b-479f-9d0f-deb2c007804a-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:10:15.568174 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:15.568000    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6a309f7-102b-479f-9d0f-deb2c007804a-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:10:15.568174 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:15.568011    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6a309f7-102b-479f-9d0f-deb2c007804a-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:10:15.568174 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:15.568019    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6a309f7-102b-479f-9d0f-deb2c007804a-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:10:16.329692 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:16.329641    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-t42ts" podUID="d6a309f7-102b-479f-9d0f-deb2c007804a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 18:10:17.080652 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:17.080615    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6a309f7-102b-479f-9d0f-deb2c007804a" path="/var/lib/kubelet/pods/d6a309f7-102b-479f-9d0f-deb2c007804a/volumes"
Sep 16 18:10:26.169858 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:26.169817    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:10:33.701835 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:33.701795    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:10:42.289882 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:42.289834    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:10:42.413248 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:42.413207    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:10:42.508184 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:42.508143    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:10:47.478942 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:47.478905    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:10:50.376308 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:10:50.376275    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:11:02.697966 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:02.697857    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:11:02.823409 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:02.823369    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:11:09.855390 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:09.855352    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:11:11.725636 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:11.725600    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:11:16.447951 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:16.447913    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:11:16.744008 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:16.743914    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:11:25.948654 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:25.948611    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:11:33.184035 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:33.183997    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:11:33.302089 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:33.302049    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:11:44.967920 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:44.967885    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd"]
Sep 16 18:11:44.968636 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:44.968438    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6a309f7-102b-479f-9d0f-deb2c007804a" containerName="main"
Sep 16 18:11:44.968636 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:44.968461    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6a309f7-102b-479f-9d0f-deb2c007804a" containerName="main"
Sep 16 18:11:44.968636 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:44.968594    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6a309f7-102b-479f-9d0f-deb2c007804a" containerName="main"
Sep 16 18:11:44.971484 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:44.971461    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd"
Sep 16 18:11:44.974938 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:44.974906    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 16 18:11:44.976036 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:44.976013    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-9t6sk\""
Sep 16 18:11:44.976155 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:44.976092    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 16 18:11:44.976155 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:44.976098    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 16 18:11:44.986515 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:44.986490    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd"]
Sep 16 18:11:45.012517 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:45.012470    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:11:45.095456 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:45.095424    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd\" (UID: \"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd"
Sep 16 18:11:45.095658 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:45.095462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd\" (UID: \"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd"
Sep 16 18:11:45.095658 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:45.095482    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd\" (UID: \"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd"
Sep 16 18:11:45.095658 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:45.095582    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p7cz\" (UniqueName: \"kubernetes.io/projected/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-kube-api-access-4p7cz\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd\" (UID: \"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd"
Sep 16 18:11:45.095658 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:45.095645    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd\" (UID: \"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd"
Sep 16 18:11:45.095859 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:45.095680    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd\" (UID: \"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd"
Sep 16 18:11:45.196509 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:45.196469    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd\" (UID: \"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd"
Sep 16 18:11:45.196509 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:45.196507    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd\" (UID: \"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd"
Sep 16 18:11:45.196726 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:45.196532    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p7cz\" (UniqueName: \"kubernetes.io/projected/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-kube-api-access-4p7cz\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd\" (UID: \"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd"
Sep 16 18:11:45.196726 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:45.196571    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd\" (UID: \"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd"
Sep 16 18:11:45.196726 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:45.196594    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd\" (UID: \"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd"
Sep 16 18:11:45.196726 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:45.196658    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd\" (UID: \"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd"
Sep 16 18:11:45.196995 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:45.196962    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd\" (UID: \"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd"
Sep 16 18:11:45.197166 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:45.197143    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd\" (UID: \"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd"
Sep 16 18:11:45.197231 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:45.197182    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd\" (UID: \"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd"
Sep 16 18:11:45.198856 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:45.198836    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd\" (UID: \"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd"
Sep 16 18:11:45.199063 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:45.199045    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd\" (UID: \"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd"
Sep 16 18:11:45.219279 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:45.219218    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p7cz\" (UniqueName: \"kubernetes.io/projected/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-kube-api-access-4p7cz\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd\" (UID: \"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd"
Sep 16 18:11:45.251917 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:45.251873    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:11:45.282627 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:45.282597    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd"
Sep 16 18:11:45.557259 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:45.557223    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd"]
Sep 16 18:11:45.558734 ip-10-0-136-122 kubenswrapper[2577]: W0916 18:11:45.558708    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea9e69c_6cfb_4cc5_91b1_fe7969a608f4.slice/crio-54db3f395fd4a974d6b987ba239eac5ab03d4bd173c716e4b83c93470168278c WatchSource:0}: Error finding container 54db3f395fd4a974d6b987ba239eac5ab03d4bd173c716e4b83c93470168278c: Status 404 returned error can't find the container with id 54db3f395fd4a974d6b987ba239eac5ab03d4bd173c716e4b83c93470168278c
Sep 16 18:11:45.941642 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:45.941601    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd" event={"ID":"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4","Type":"ContainerStarted","Data":"aaaf5c7f67f3e5e9d7b8f77d57b9235f733e3c3a63fbc5a9dbd60ca3112169fb"}
Sep 16 18:11:45.941890 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:45.941656    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd" event={"ID":"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4","Type":"ContainerStarted","Data":"54db3f395fd4a974d6b987ba239eac5ab03d4bd173c716e4b83c93470168278c"}
Sep 16 18:11:45.965546 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:45.965500    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd" podStartSLOduration=1.965480557 podStartE2EDuration="1.965480557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:11:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:11:45.962201599 +0000 UTC m=+2827.324880385" watchObservedRunningTime="2026-09-16 18:11:45.965480557 +0000 UTC m=+2827.328159372"
Sep 16 18:11:46.282761 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:46.282668    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd"
Sep 16 18:11:55.283571 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:55.283532    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd"
Sep 16 18:11:55.296123 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:55.296098    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd"
Sep 16 18:11:56.004925 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:11:56.004896    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd"
Sep 16 18:12:28.245659 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:28.245618    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:12:36.139461 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:36.139379    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp"]
Sep 16 18:12:36.142919 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:36.142898    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp"
Sep 16 18:12:36.148767 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:36.148742    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 16 18:12:36.163366 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:36.163340    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp"]
Sep 16 18:12:36.168301 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:36.168278    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:12:36.233770 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:36.233732    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwfdr\" (UniqueName: \"kubernetes.io/projected/a657fb9f-f963-4320-aff4-90fbb94f7a3b-kube-api-access-hwfdr\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp\" (UID: \"a657fb9f-f963-4320-aff4-90fbb94f7a3b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp"
Sep 16 18:12:36.233963 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:36.233782    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a657fb9f-f963-4320-aff4-90fbb94f7a3b-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp\" (UID: \"a657fb9f-f963-4320-aff4-90fbb94f7a3b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp"
Sep 16 18:12:36.233963 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:36.233884    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a657fb9f-f963-4320-aff4-90fbb94f7a3b-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp\" (UID: \"a657fb9f-f963-4320-aff4-90fbb94f7a3b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp"
Sep 16 18:12:36.233963 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:36.233920    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a657fb9f-f963-4320-aff4-90fbb94f7a3b-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp\" (UID: \"a657fb9f-f963-4320-aff4-90fbb94f7a3b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp"
Sep 16 18:12:36.233963 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:36.233957    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a657fb9f-f963-4320-aff4-90fbb94f7a3b-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp\" (UID: \"a657fb9f-f963-4320-aff4-90fbb94f7a3b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp"
Sep 16 18:12:36.234102 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:36.233999    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a657fb9f-f963-4320-aff4-90fbb94f7a3b-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp\" (UID: \"a657fb9f-f963-4320-aff4-90fbb94f7a3b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp"
Sep 16 18:12:36.335468 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:36.335424    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwfdr\" (UniqueName: \"kubernetes.io/projected/a657fb9f-f963-4320-aff4-90fbb94f7a3b-kube-api-access-hwfdr\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp\" (UID: \"a657fb9f-f963-4320-aff4-90fbb94f7a3b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp"
Sep 16 18:12:36.335658 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:36.335484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a657fb9f-f963-4320-aff4-90fbb94f7a3b-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp\" (UID: \"a657fb9f-f963-4320-aff4-90fbb94f7a3b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp"
Sep 16 18:12:36.335658 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:36.335533    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a657fb9f-f963-4320-aff4-90fbb94f7a3b-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp\" (UID: \"a657fb9f-f963-4320-aff4-90fbb94f7a3b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp"
Sep 16 18:12:36.335658 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:36.335555    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a657fb9f-f963-4320-aff4-90fbb94f7a3b-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp\" (UID: \"a657fb9f-f963-4320-aff4-90fbb94f7a3b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp"
Sep 16 18:12:36.335658 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:36.335585    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a657fb9f-f963-4320-aff4-90fbb94f7a3b-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp\" (UID: \"a657fb9f-f963-4320-aff4-90fbb94f7a3b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp"
Sep 16 18:12:36.335658 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:36.335635    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a657fb9f-f963-4320-aff4-90fbb94f7a3b-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp\" (UID: \"a657fb9f-f963-4320-aff4-90fbb94f7a3b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp"
Sep 16 18:12:36.336054 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:36.336035    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a657fb9f-f963-4320-aff4-90fbb94f7a3b-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp\" (UID: \"a657fb9f-f963-4320-aff4-90fbb94f7a3b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp"
Sep 16 18:12:36.336122 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:36.336086    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a657fb9f-f963-4320-aff4-90fbb94f7a3b-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp\" (UID: \"a657fb9f-f963-4320-aff4-90fbb94f7a3b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp"
Sep 16 18:12:36.336181 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:36.336089    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a657fb9f-f963-4320-aff4-90fbb94f7a3b-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp\" (UID: \"a657fb9f-f963-4320-aff4-90fbb94f7a3b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp"
Sep 16 18:12:36.337929 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:36.337904    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a657fb9f-f963-4320-aff4-90fbb94f7a3b-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp\" (UID: \"a657fb9f-f963-4320-aff4-90fbb94f7a3b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp"
Sep 16 18:12:36.338062 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:36.338045    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a657fb9f-f963-4320-aff4-90fbb94f7a3b-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp\" (UID: \"a657fb9f-f963-4320-aff4-90fbb94f7a3b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp"
Sep 16 18:12:36.344245 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:36.344220    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwfdr\" (UniqueName: \"kubernetes.io/projected/a657fb9f-f963-4320-aff4-90fbb94f7a3b-kube-api-access-hwfdr\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp\" (UID: \"a657fb9f-f963-4320-aff4-90fbb94f7a3b\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp"
Sep 16 18:12:36.453229 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:36.453141    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp"
Sep 16 18:12:36.522955 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:36.522897    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:12:36.739818 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:36.739791    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp"]
Sep 16 18:12:36.742144 ip-10-0-136-122 kubenswrapper[2577]: W0916 18:12:36.742114    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda657fb9f_f963_4320_aff4_90fbb94f7a3b.slice/crio-445c7bf6399dcd5f71e31514e072e8879496c665e6b3277e082ff66614e98b43 WatchSource:0}: Error finding container 445c7bf6399dcd5f71e31514e072e8879496c665e6b3277e082ff66614e98b43: Status 404 returned error can't find the container with id 445c7bf6399dcd5f71e31514e072e8879496c665e6b3277e082ff66614e98b43
Sep 16 18:12:37.216341 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:37.216301    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp" event={"ID":"a657fb9f-f963-4320-aff4-90fbb94f7a3b","Type":"ContainerStarted","Data":"abb915f3a9042bc7de0201e89a65eb12444f885b4e01f9919ea012ed4740e348"}
Sep 16 18:12:37.216819 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:37.216365    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp" event={"ID":"a657fb9f-f963-4320-aff4-90fbb94f7a3b","Type":"ContainerStarted","Data":"445c7bf6399dcd5f71e31514e072e8879496c665e6b3277e082ff66614e98b43"}
Sep 16 18:12:37.242813 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:37.242748    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp" podStartSLOduration=1.242731386 podStartE2EDuration="1.242731386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:12:37.240318476 +0000 UTC m=+2878.602997263" watchObservedRunningTime="2026-09-16 18:12:37.242731386 +0000 UTC m=+2878.605410204"
Sep 16 18:12:37.454218 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:37.454181    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp"
Sep 16 18:12:46.453411 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:46.453373    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp"
Sep 16 18:12:46.466059 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:46.466029    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp"
Sep 16 18:12:47.280297 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:47.280263    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp"
Sep 16 18:12:51.100170 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:51.100127    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:12:52.439934 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:12:52.439878    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:13:09.578667 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:09.578631    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp"]
Sep 16 18:13:09.587884 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:13:09.587827    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 18:13:09.587884 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:13:09.587884 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:13:09.587884 ip-10-0-136-122 kubenswrapper[2577]:  > containerID="abb915f3a9042bc7de0201e89a65eb12444f885b4e01f9919ea012ed4740e348" cmd=["/bin/sleep","15"]
Sep 16 18:13:09.588193 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:09.587900    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp" message="" err=<
Sep 16 18:13:09.588193 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:13:09.588193 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:13:09.588193 ip-10-0-136-122 kubenswrapper[2577]:  >
Sep 16 18:13:09.588193 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:13:09.587918    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 18:13:09.588193 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:13:09.588193 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:13:09.588193 ip-10-0-136-122 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp" podUID="a657fb9f-f963-4320-aff4-90fbb94f7a3b" containerName="main" containerID="cri-o://abb915f3a9042bc7de0201e89a65eb12444f885b4e01f9919ea012ed4740e348"
Sep 16 18:13:09.588193 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:09.587971    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp" podUID="a657fb9f-f963-4320-aff4-90fbb94f7a3b" containerName="main" containerID="cri-o://abb915f3a9042bc7de0201e89a65eb12444f885b4e01f9919ea012ed4740e348" gracePeriod=60
Sep 16 18:13:09.719117 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:09.719088    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:13:09.785179 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:09.785140    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:13:09.941219 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:09.941191    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp"
Sep 16 18:13:10.040753 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:10.040711    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a657fb9f-f963-4320-aff4-90fbb94f7a3b-home\") pod \"a657fb9f-f963-4320-aff4-90fbb94f7a3b\" (UID: \"a657fb9f-f963-4320-aff4-90fbb94f7a3b\") "
Sep 16 18:13:10.040753 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:10.040755    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a657fb9f-f963-4320-aff4-90fbb94f7a3b-tls-certs\") pod \"a657fb9f-f963-4320-aff4-90fbb94f7a3b\" (UID: \"a657fb9f-f963-4320-aff4-90fbb94f7a3b\") "
Sep 16 18:13:10.041033 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:10.040782    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwfdr\" (UniqueName: \"kubernetes.io/projected/a657fb9f-f963-4320-aff4-90fbb94f7a3b-kube-api-access-hwfdr\") pod \"a657fb9f-f963-4320-aff4-90fbb94f7a3b\" (UID: \"a657fb9f-f963-4320-aff4-90fbb94f7a3b\") "
Sep 16 18:13:10.041033 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:10.040804    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a657fb9f-f963-4320-aff4-90fbb94f7a3b-tmp-dir\") pod \"a657fb9f-f963-4320-aff4-90fbb94f7a3b\" (UID: \"a657fb9f-f963-4320-aff4-90fbb94f7a3b\") "
Sep 16 18:13:10.041033 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:10.040831    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a657fb9f-f963-4320-aff4-90fbb94f7a3b-model-cache\") pod \"a657fb9f-f963-4320-aff4-90fbb94f7a3b\" (UID: \"a657fb9f-f963-4320-aff4-90fbb94f7a3b\") "
Sep 16 18:13:10.041033 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:10.040858    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a657fb9f-f963-4320-aff4-90fbb94f7a3b-dshm\") pod \"a657fb9f-f963-4320-aff4-90fbb94f7a3b\" (UID: \"a657fb9f-f963-4320-aff4-90fbb94f7a3b\") "
Sep 16 18:13:10.041308 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:10.041098    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a657fb9f-f963-4320-aff4-90fbb94f7a3b-model-cache" (OuterVolumeSpecName: "model-cache") pod "a657fb9f-f963-4320-aff4-90fbb94f7a3b" (UID: "a657fb9f-f963-4320-aff4-90fbb94f7a3b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:13:10.041308 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:10.041121    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a657fb9f-f963-4320-aff4-90fbb94f7a3b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a657fb9f-f963-4320-aff4-90fbb94f7a3b" (UID: "a657fb9f-f963-4320-aff4-90fbb94f7a3b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:13:10.041308 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:10.041284    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a657fb9f-f963-4320-aff4-90fbb94f7a3b-home" (OuterVolumeSpecName: "home") pod "a657fb9f-f963-4320-aff4-90fbb94f7a3b" (UID: "a657fb9f-f963-4320-aff4-90fbb94f7a3b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:13:10.043228 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:10.043203    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a657fb9f-f963-4320-aff4-90fbb94f7a3b-kube-api-access-hwfdr" (OuterVolumeSpecName: "kube-api-access-hwfdr") pod "a657fb9f-f963-4320-aff4-90fbb94f7a3b" (UID: "a657fb9f-f963-4320-aff4-90fbb94f7a3b"). InnerVolumeSpecName "kube-api-access-hwfdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:13:10.043357 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:10.043241    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a657fb9f-f963-4320-aff4-90fbb94f7a3b-dshm" (OuterVolumeSpecName: "dshm") pod "a657fb9f-f963-4320-aff4-90fbb94f7a3b" (UID: "a657fb9f-f963-4320-aff4-90fbb94f7a3b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:13:10.043357 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:10.043243    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a657fb9f-f963-4320-aff4-90fbb94f7a3b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a657fb9f-f963-4320-aff4-90fbb94f7a3b" (UID: "a657fb9f-f963-4320-aff4-90fbb94f7a3b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:13:10.141896 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:10.141794    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a657fb9f-f963-4320-aff4-90fbb94f7a3b-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:13:10.141896 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:10.141825    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a657fb9f-f963-4320-aff4-90fbb94f7a3b-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:13:10.141896 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:10.141837    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwfdr\" (UniqueName: \"kubernetes.io/projected/a657fb9f-f963-4320-aff4-90fbb94f7a3b-kube-api-access-hwfdr\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:13:10.141896 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:10.141846    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a657fb9f-f963-4320-aff4-90fbb94f7a3b-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:13:10.141896 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:10.141855    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a657fb9f-f963-4320-aff4-90fbb94f7a3b-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:13:10.141896 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:10.141875    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a657fb9f-f963-4320-aff4-90fbb94f7a3b-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:13:10.398084 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:10.397998    2577 generic.go:356] "Generic (PLEG): container finished" podID="a657fb9f-f963-4320-aff4-90fbb94f7a3b" containerID="abb915f3a9042bc7de0201e89a65eb12444f885b4e01f9919ea012ed4740e348" exitCode=0
Sep 16 18:13:10.398084 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:10.398063    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp"
Sep 16 18:13:10.398270 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:10.398074    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp" event={"ID":"a657fb9f-f963-4320-aff4-90fbb94f7a3b","Type":"ContainerDied","Data":"abb915f3a9042bc7de0201e89a65eb12444f885b4e01f9919ea012ed4740e348"}
Sep 16 18:13:10.398270 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:10.398128    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp" event={"ID":"a657fb9f-f963-4320-aff4-90fbb94f7a3b","Type":"ContainerDied","Data":"445c7bf6399dcd5f71e31514e072e8879496c665e6b3277e082ff66614e98b43"}
Sep 16 18:13:10.398270 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:10.398151    2577 scope.go:117] "RemoveContainer" containerID="abb915f3a9042bc7de0201e89a65eb12444f885b4e01f9919ea012ed4740e348"
Sep 16 18:13:10.408041 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:10.408024    2577 scope.go:117] "RemoveContainer" containerID="abb915f3a9042bc7de0201e89a65eb12444f885b4e01f9919ea012ed4740e348"
Sep 16 18:13:10.408297 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:13:10.408278    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abb915f3a9042bc7de0201e89a65eb12444f885b4e01f9919ea012ed4740e348\": container with ID starting with abb915f3a9042bc7de0201e89a65eb12444f885b4e01f9919ea012ed4740e348 not found: ID does not exist" containerID="abb915f3a9042bc7de0201e89a65eb12444f885b4e01f9919ea012ed4740e348"
Sep 16 18:13:10.408356 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:10.408305    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abb915f3a9042bc7de0201e89a65eb12444f885b4e01f9919ea012ed4740e348"} err="failed to get container status \"abb915f3a9042bc7de0201e89a65eb12444f885b4e01f9919ea012ed4740e348\": rpc error: code = NotFound desc = could not find container \"abb915f3a9042bc7de0201e89a65eb12444f885b4e01f9919ea012ed4740e348\": container with ID starting with abb915f3a9042bc7de0201e89a65eb12444f885b4e01f9919ea012ed4740e348 not found: ID does not exist"
Sep 16 18:13:10.423208 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:10.423179    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp"]
Sep 16 18:13:10.426559 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:10.426534    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ct4cp"]
Sep 16 18:13:11.082145 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:11.082110    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a657fb9f-f963-4320-aff4-90fbb94f7a3b" path="/var/lib/kubelet/pods/a657fb9f-f963-4320-aff4-90fbb94f7a3b/volumes"
Sep 16 18:13:37.879754 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:37.879715    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:13:39.524074 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:39.524036    2577 scope.go:117] "RemoveContainer" containerID="979d10049d122180cb79fe8bc72a439c3f97aa98017de2809327f5b2a0a28c03"
Sep 16 18:13:39.532694 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:39.532667    2577 scope.go:117] "RemoveContainer" containerID="05c23dcdd6254f6afb4aea10368ae7d3034132bc29ff55c91d11fffe3457915a"
Sep 16 18:13:41.086109 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:41.086072    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:13:41.200774 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:41.200725    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:13:41.307984 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:41.307944    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:13:41.803343 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:41.803306    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:13:46.149425 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:46.149385    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:13:56.512050 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:13:56.512004    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:14:20.020411 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:20.020367    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:14:21.173810 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:21.173771    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:14:21.182504 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:21.182469    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd"]
Sep 16 18:14:21.191742 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:14:21.191708    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 18:14:21.191742 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:14:21.191742 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:14:21.191742 ip-10-0-136-122 kubenswrapper[2577]:  > containerID="aaaf5c7f67f3e5e9d7b8f77d57b9235f733e3c3a63fbc5a9dbd60ca3112169fb" cmd=["/bin/sleep","15"]
Sep 16 18:14:21.192261 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:21.191756    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd" message="" err=<
Sep 16 18:14:21.192261 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:14:21.192261 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:14:21.192261 ip-10-0-136-122 kubenswrapper[2577]:  >
Sep 16 18:14:21.192261 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:14:21.191770    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 18:14:21.192261 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:14:21.192261 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:14:21.192261 ip-10-0-136-122 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd" podUID="fea9e69c-6cfb-4cc5-91b1-fe7969a608f4" containerName="main" containerID="cri-o://aaaf5c7f67f3e5e9d7b8f77d57b9235f733e3c3a63fbc5a9dbd60ca3112169fb"
Sep 16 18:14:21.192261 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:21.191814    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd" podUID="fea9e69c-6cfb-4cc5-91b1-fe7969a608f4" containerName="main" containerID="cri-o://aaaf5c7f67f3e5e9d7b8f77d57b9235f733e3c3a63fbc5a9dbd60ca3112169fb" gracePeriod=60
Sep 16 18:14:21.273941 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:21.273896    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:14:21.776965 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:21.776929    2577 generic.go:356] "Generic (PLEG): container finished" podID="fea9e69c-6cfb-4cc5-91b1-fe7969a608f4" containerID="aaaf5c7f67f3e5e9d7b8f77d57b9235f733e3c3a63fbc5a9dbd60ca3112169fb" exitCode=0
Sep 16 18:14:21.777137 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:21.777007    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd" event={"ID":"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4","Type":"ContainerDied","Data":"aaaf5c7f67f3e5e9d7b8f77d57b9235f733e3c3a63fbc5a9dbd60ca3112169fb"}
Sep 16 18:14:21.944526 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:21.944500    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd"
Sep 16 18:14:22.067111 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:22.067085    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4p7cz\" (UniqueName: \"kubernetes.io/projected/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-kube-api-access-4p7cz\") pod \"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4\" (UID: \"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4\") "
Sep 16 18:14:22.067263 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:22.067119    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-model-cache\") pod \"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4\" (UID: \"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4\") "
Sep 16 18:14:22.067263 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:22.067142    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-home\") pod \"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4\" (UID: \"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4\") "
Sep 16 18:14:22.067263 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:22.067169    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-tmp-dir\") pod \"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4\" (UID: \"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4\") "
Sep 16 18:14:22.067263 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:22.067203    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-dshm\") pod \"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4\" (UID: \"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4\") "
Sep 16 18:14:22.067263 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:22.067221    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-tls-certs\") pod \"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4\" (UID: \"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4\") "
Sep 16 18:14:22.067534 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:22.067398    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-model-cache" (OuterVolumeSpecName: "model-cache") pod "fea9e69c-6cfb-4cc5-91b1-fe7969a608f4" (UID: "fea9e69c-6cfb-4cc5-91b1-fe7969a608f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:14:22.067534 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:22.067416    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fea9e69c-6cfb-4cc5-91b1-fe7969a608f4" (UID: "fea9e69c-6cfb-4cc5-91b1-fe7969a608f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:14:22.067534 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:22.067468    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-home" (OuterVolumeSpecName: "home") pod "fea9e69c-6cfb-4cc5-91b1-fe7969a608f4" (UID: "fea9e69c-6cfb-4cc5-91b1-fe7969a608f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:14:22.069302 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:22.069272    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-dshm" (OuterVolumeSpecName: "dshm") pod "fea9e69c-6cfb-4cc5-91b1-fe7969a608f4" (UID: "fea9e69c-6cfb-4cc5-91b1-fe7969a608f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:14:22.069403 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:22.069318    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fea9e69c-6cfb-4cc5-91b1-fe7969a608f4" (UID: "fea9e69c-6cfb-4cc5-91b1-fe7969a608f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:14:22.069403 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:22.069369    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-kube-api-access-4p7cz" (OuterVolumeSpecName: "kube-api-access-4p7cz") pod "fea9e69c-6cfb-4cc5-91b1-fe7969a608f4" (UID: "fea9e69c-6cfb-4cc5-91b1-fe7969a608f4"). InnerVolumeSpecName "kube-api-access-4p7cz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:14:22.168182 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:22.168149    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p7cz\" (UniqueName: \"kubernetes.io/projected/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-kube-api-access-4p7cz\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:14:22.168182 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:22.168178    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:14:22.168182 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:22.168187    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:14:22.168403 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:22.168229    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:14:22.168403 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:22.168238    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:14:22.168403 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:22.168245    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:14:22.784190 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:22.784160    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd"
Sep 16 18:14:22.784605 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:22.784159    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd" event={"ID":"fea9e69c-6cfb-4cc5-91b1-fe7969a608f4","Type":"ContainerDied","Data":"54db3f395fd4a974d6b987ba239eac5ab03d4bd173c716e4b83c93470168278c"}
Sep 16 18:14:22.784605 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:22.784304    2577 scope.go:117] "RemoveContainer" containerID="aaaf5c7f67f3e5e9d7b8f77d57b9235f733e3c3a63fbc5a9dbd60ca3112169fb"
Sep 16 18:14:22.822390 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:22.822358    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd"]
Sep 16 18:14:22.831641 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:22.831616    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-8lrqd"]
Sep 16 18:14:23.080255 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:23.080221    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fea9e69c-6cfb-4cc5-91b1-fe7969a608f4" path="/var/lib/kubelet/pods/fea9e69c-6cfb-4cc5-91b1-fe7969a608f4/volumes"
Sep 16 18:14:24.725149 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:24.725116    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:14:31.175062 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:31.175022    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:14:31.299132 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:14:31.299087    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:15:03.708375 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:03.708335    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:15:10.043766 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:10.043733    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:15:10.151480 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:10.151445    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:15:15.472824 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.472787    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf"]
Sep 16 18:15:15.473198 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.473146    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fea9e69c-6cfb-4cc5-91b1-fe7969a608f4" containerName="main"
Sep 16 18:15:15.473198 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.473159    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fea9e69c-6cfb-4cc5-91b1-fe7969a608f4" containerName="main"
Sep 16 18:15:15.473198 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.473170    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a657fb9f-f963-4320-aff4-90fbb94f7a3b" containerName="main"
Sep 16 18:15:15.473198 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.473175    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a657fb9f-f963-4320-aff4-90fbb94f7a3b" containerName="main"
Sep 16 18:15:15.473348 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.473240    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a657fb9f-f963-4320-aff4-90fbb94f7a3b" containerName="main"
Sep 16 18:15:15.473348 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.473249    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fea9e69c-6cfb-4cc5-91b1-fe7969a608f4" containerName="main"
Sep 16 18:15:15.475976 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.475955    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf"
Sep 16 18:15:15.482774 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.482245    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 16 18:15:15.482774 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.482255    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 16 18:15:15.482774 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.482255    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 16 18:15:15.482774 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.482245    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-9bp9x\""
Sep 16 18:15:15.490596 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.490575    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf"]
Sep 16 18:15:15.509062 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.509034    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf\" (UID: \"9e4dc259-a259-42f3-9fc1-74c4e9d45adc\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf"
Sep 16 18:15:15.509181 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.509080    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf\" (UID: \"9e4dc259-a259-42f3-9fc1-74c4e9d45adc\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf"
Sep 16 18:15:15.509181 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.509152    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf\" (UID: \"9e4dc259-a259-42f3-9fc1-74c4e9d45adc\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf"
Sep 16 18:15:15.509255 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.509186    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2jhv\" (UniqueName: \"kubernetes.io/projected/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-kube-api-access-d2jhv\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf\" (UID: \"9e4dc259-a259-42f3-9fc1-74c4e9d45adc\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf"
Sep 16 18:15:15.509255 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.509235    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf\" (UID: \"9e4dc259-a259-42f3-9fc1-74c4e9d45adc\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf"
Sep 16 18:15:15.509323 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.509256    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf\" (UID: \"9e4dc259-a259-42f3-9fc1-74c4e9d45adc\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf"
Sep 16 18:15:15.514997 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.514968    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:15:15.609967 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.609928    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf\" (UID: \"9e4dc259-a259-42f3-9fc1-74c4e9d45adc\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf"
Sep 16 18:15:15.609967 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.609968    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2jhv\" (UniqueName: \"kubernetes.io/projected/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-kube-api-access-d2jhv\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf\" (UID: \"9e4dc259-a259-42f3-9fc1-74c4e9d45adc\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf"
Sep 16 18:15:15.610181 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.610029    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf\" (UID: \"9e4dc259-a259-42f3-9fc1-74c4e9d45adc\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf"
Sep 16 18:15:15.610181 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.610054    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf\" (UID: \"9e4dc259-a259-42f3-9fc1-74c4e9d45adc\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf"
Sep 16 18:15:15.610181 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.610094    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf\" (UID: \"9e4dc259-a259-42f3-9fc1-74c4e9d45adc\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf"
Sep 16 18:15:15.610181 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.610125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf\" (UID: \"9e4dc259-a259-42f3-9fc1-74c4e9d45adc\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf"
Sep 16 18:15:15.610423 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.610402    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf\" (UID: \"9e4dc259-a259-42f3-9fc1-74c4e9d45adc\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf"
Sep 16 18:15:15.610549 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.610462    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf\" (UID: \"9e4dc259-a259-42f3-9fc1-74c4e9d45adc\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf"
Sep 16 18:15:15.610549 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.610514    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf\" (UID: \"9e4dc259-a259-42f3-9fc1-74c4e9d45adc\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf"
Sep 16 18:15:15.612308 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.612287    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf\" (UID: \"9e4dc259-a259-42f3-9fc1-74c4e9d45adc\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf"
Sep 16 18:15:15.612714 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.612690    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf\" (UID: \"9e4dc259-a259-42f3-9fc1-74c4e9d45adc\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf"
Sep 16 18:15:15.619105 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.619085    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2jhv\" (UniqueName: \"kubernetes.io/projected/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-kube-api-access-d2jhv\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf\" (UID: \"9e4dc259-a259-42f3-9fc1-74c4e9d45adc\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf"
Sep 16 18:15:15.780381 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.780300    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:15:15.786090 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:15.786060    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf"
Sep 16 18:15:16.063246 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:16.063217    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf"]
Sep 16 18:15:16.064793 ip-10-0-136-122 kubenswrapper[2577]: W0916 18:15:16.064763    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e4dc259_a259_42f3_9fc1_74c4e9d45adc.slice/crio-6e2e7cdbfe889d2895f84f7c8f32f18bd47f558bf78fd8aef57487350135eb5c WatchSource:0}: Error finding container 6e2e7cdbfe889d2895f84f7c8f32f18bd47f558bf78fd8aef57487350135eb5c: Status 404 returned error can't find the container with id 6e2e7cdbfe889d2895f84f7c8f32f18bd47f558bf78fd8aef57487350135eb5c
Sep 16 18:15:17.080134 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:17.080102    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf" event={"ID":"9e4dc259-a259-42f3-9fc1-74c4e9d45adc","Type":"ContainerStarted","Data":"ed94d57fdc7a0ae67252e557afdb17f289f27a905e196201228acc362963b05d"}
Sep 16 18:15:17.080497 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:17.080152    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf" event={"ID":"9e4dc259-a259-42f3-9fc1-74c4e9d45adc","Type":"ContainerStarted","Data":"6e2e7cdbfe889d2895f84f7c8f32f18bd47f558bf78fd8aef57487350135eb5c"}
Sep 16 18:15:17.101586 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:17.101536    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf" podStartSLOduration=2.101521224 podStartE2EDuration="2.101521224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:15:17.098939851 +0000 UTC m=+3038.461618644" watchObservedRunningTime="2026-09-16 18:15:17.101521224 +0000 UTC m=+3038.464200043"
Sep 16 18:15:17.787178 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:17.787139    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf"
Sep 16 18:15:18.062881 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:18.062833    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:15:18.518228 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:18.518145    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:15:24.527231 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:24.527188    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:15:24.631537 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:24.631502    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:15:25.786878 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:25.786831    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf"
Sep 16 18:15:25.799061 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:25.799034    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf"
Sep 16 18:15:26.129679 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:26.129644    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf"
Sep 16 18:15:28.898574 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:28.898534    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:15:39.684590 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:39.684537    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:15:39.786723 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:39.786687    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:15:40.060813 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.060778    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"]
Sep 16 18:15:40.064230 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.064209    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:15:40.066896 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.066858    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 16 18:15:40.067085 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.067063    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Sep 16 18:15:40.067174 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.067121    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-b2v5r\""
Sep 16 18:15:40.067945 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.067926    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 16 18:15:40.080461 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.080432    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:15:40.082773 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.082750    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"]
Sep 16 18:15:40.165386 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.165346    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:15:40.222993 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.222952    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07035df6-6670-48b6-bb28-09c19f657aae-istio-podinfo\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-s48rg\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:15:40.223170 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.223008    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07035df6-6670-48b6-bb28-09c19f657aae-istio-envoy\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-s48rg\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:15:40.223170 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.223047    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07035df6-6670-48b6-bb28-09c19f657aae-workload-certs\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-s48rg\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:15:40.223170 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.223092    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcvgm\" (UniqueName: \"kubernetes.io/projected/07035df6-6670-48b6-bb28-09c19f657aae-kube-api-access-vcvgm\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-s48rg\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:15:40.223170 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.223116    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07035df6-6670-48b6-bb28-09c19f657aae-istio-data\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-s48rg\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:15:40.223170 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.223142    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07035df6-6670-48b6-bb28-09c19f657aae-workload-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-s48rg\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:15:40.223170 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.223156    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07035df6-6670-48b6-bb28-09c19f657aae-istio-token\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-s48rg\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:15:40.223400 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.223178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07035df6-6670-48b6-bb28-09c19f657aae-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-s48rg\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:15:40.223400 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.223218    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07035df6-6670-48b6-bb28-09c19f657aae-credential-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-s48rg\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:15:40.323997 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.323963    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcvgm\" (UniqueName: \"kubernetes.io/projected/07035df6-6670-48b6-bb28-09c19f657aae-kube-api-access-vcvgm\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-s48rg\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:15:40.324158 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.324045    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07035df6-6670-48b6-bb28-09c19f657aae-istio-data\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-s48rg\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:15:40.324158 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.324112    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07035df6-6670-48b6-bb28-09c19f657aae-workload-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-s48rg\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:15:40.324158 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.324138    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07035df6-6670-48b6-bb28-09c19f657aae-istio-token\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-s48rg\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:15:40.324268 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.324159    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07035df6-6670-48b6-bb28-09c19f657aae-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-s48rg\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:15:40.324268 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.324218    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07035df6-6670-48b6-bb28-09c19f657aae-credential-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-s48rg\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:15:40.324268 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.324256    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07035df6-6670-48b6-bb28-09c19f657aae-istio-podinfo\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-s48rg\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:15:40.324419 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.324311    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07035df6-6670-48b6-bb28-09c19f657aae-istio-envoy\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-s48rg\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:15:40.324419 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.324344    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07035df6-6670-48b6-bb28-09c19f657aae-workload-certs\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-s48rg\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:15:40.324419 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.324400    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07035df6-6670-48b6-bb28-09c19f657aae-istio-data\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-s48rg\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:15:40.324581 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.324495    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07035df6-6670-48b6-bb28-09c19f657aae-workload-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-s48rg\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:15:40.324698 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.324674    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07035df6-6670-48b6-bb28-09c19f657aae-credential-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-s48rg\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:15:40.324757 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.324701    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07035df6-6670-48b6-bb28-09c19f657aae-workload-certs\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-s48rg\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:15:40.324966 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.324946    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07035df6-6670-48b6-bb28-09c19f657aae-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-s48rg\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:15:40.326661 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.326635    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07035df6-6670-48b6-bb28-09c19f657aae-istio-envoy\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-s48rg\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:15:40.326854 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.326838    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07035df6-6670-48b6-bb28-09c19f657aae-istio-podinfo\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-s48rg\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:15:40.332233 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.332208    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07035df6-6670-48b6-bb28-09c19f657aae-istio-token\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-s48rg\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:15:40.332390 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.332370    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcvgm\" (UniqueName: \"kubernetes.io/projected/07035df6-6670-48b6-bb28-09c19f657aae-kube-api-access-vcvgm\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-s48rg\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:15:40.374375 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.374348    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:15:40.657253 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.657222    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"]
Sep 16 18:15:40.658963 ip-10-0-136-122 kubenswrapper[2577]: W0916 18:15:40.658934    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07035df6_6670_48b6_bb28_09c19f657aae.slice/crio-b8e0e04fa6a57bab06f2236188365e933a6f0301fa41af798d742625f53eee44 WatchSource:0}: Error finding container b8e0e04fa6a57bab06f2236188365e933a6f0301fa41af798d742625f53eee44: Status 404 returned error can't find the container with id b8e0e04fa6a57bab06f2236188365e933a6f0301fa41af798d742625f53eee44
Sep 16 18:15:40.661179 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:40.661157    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:15:41.204133 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:41.204092    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg" event={"ID":"07035df6-6670-48b6-bb28-09c19f657aae","Type":"ContainerStarted","Data":"b8e0e04fa6a57bab06f2236188365e933a6f0301fa41af798d742625f53eee44"}
Sep 16 18:15:42.051745 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:42.051665    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:15:43.581165 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:43.581122    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:15:43.581443 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:43.581201    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:15:43.581443 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:43.581239    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:15:43.644702 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:43.644667    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:15:44.224110 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:44.224069    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg" event={"ID":"07035df6-6670-48b6-bb28-09c19f657aae","Type":"ContainerStarted","Data":"78cd3c84bf4e8170fce6238aa86072a4880c46f7aa6dd3d9c44b5eb30538bb58"}
Sep 16 18:15:44.249122 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:44.249066    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg" podStartSLOduration=1.329505578 podStartE2EDuration="4.249051684s" totalImagesPullingTime="2.919546106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:15:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:15:44.24593932 +0000 UTC m=+3065.608618102" watchObservedRunningTime="2026-09-16 18:15:44.249051684 +0000 UTC m=+3065.611730469"
Sep 16 18:15:44.375528 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:44.375490    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:15:44.380554 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:44.380531    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:15:44.725630 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:44.725598    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:15:45.229107 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:45.229075    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:15:45.230215 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:45.230192    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:15:51.947951 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:51.947913    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq"]
Sep 16 18:15:51.951815 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:51.951793    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq"
Sep 16 18:15:51.958663 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:51.958641    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-hfq6s\""
Sep 16 18:15:51.958944 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:51.958919    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Sep 16 18:15:51.961920 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:51.961892    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq"]
Sep 16 18:15:51.975419 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:51.975392    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:15:52.141874 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:52.141826    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e732fc41-9e6d-4270-ba76-88c22c5df822-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq\" (UID: \"e732fc41-9e6d-4270-ba76-88c22c5df822\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq"
Sep 16 18:15:52.142060 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:52.141894    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt9xs\" (UniqueName: \"kubernetes.io/projected/e732fc41-9e6d-4270-ba76-88c22c5df822-kube-api-access-rt9xs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq\" (UID: \"e732fc41-9e6d-4270-ba76-88c22c5df822\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq"
Sep 16 18:15:52.142060 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:52.141953    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e732fc41-9e6d-4270-ba76-88c22c5df822-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq\" (UID: \"e732fc41-9e6d-4270-ba76-88c22c5df822\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq"
Sep 16 18:15:52.142060 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:52.142021    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e732fc41-9e6d-4270-ba76-88c22c5df822-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq\" (UID: \"e732fc41-9e6d-4270-ba76-88c22c5df822\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq"
Sep 16 18:15:52.142199 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:52.142060    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e732fc41-9e6d-4270-ba76-88c22c5df822-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq\" (UID: \"e732fc41-9e6d-4270-ba76-88c22c5df822\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq"
Sep 16 18:15:52.142199 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:52.142118    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e732fc41-9e6d-4270-ba76-88c22c5df822-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq\" (UID: \"e732fc41-9e6d-4270-ba76-88c22c5df822\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq"
Sep 16 18:15:52.243697 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:52.243615    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e732fc41-9e6d-4270-ba76-88c22c5df822-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq\" (UID: \"e732fc41-9e6d-4270-ba76-88c22c5df822\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq"
Sep 16 18:15:52.243697 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:52.243659    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt9xs\" (UniqueName: \"kubernetes.io/projected/e732fc41-9e6d-4270-ba76-88c22c5df822-kube-api-access-rt9xs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq\" (UID: \"e732fc41-9e6d-4270-ba76-88c22c5df822\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq"
Sep 16 18:15:52.243934 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:52.243707    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e732fc41-9e6d-4270-ba76-88c22c5df822-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq\" (UID: \"e732fc41-9e6d-4270-ba76-88c22c5df822\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq"
Sep 16 18:15:52.243934 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:52.243730    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e732fc41-9e6d-4270-ba76-88c22c5df822-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq\" (UID: \"e732fc41-9e6d-4270-ba76-88c22c5df822\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq"
Sep 16 18:15:52.243934 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:52.243759    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e732fc41-9e6d-4270-ba76-88c22c5df822-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq\" (UID: \"e732fc41-9e6d-4270-ba76-88c22c5df822\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq"
Sep 16 18:15:52.243934 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:52.243786    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e732fc41-9e6d-4270-ba76-88c22c5df822-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq\" (UID: \"e732fc41-9e6d-4270-ba76-88c22c5df822\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq"
Sep 16 18:15:52.244202 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:52.244175    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e732fc41-9e6d-4270-ba76-88c22c5df822-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq\" (UID: \"e732fc41-9e6d-4270-ba76-88c22c5df822\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq"
Sep 16 18:15:52.244327 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:52.244202    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e732fc41-9e6d-4270-ba76-88c22c5df822-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq\" (UID: \"e732fc41-9e6d-4270-ba76-88c22c5df822\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq"
Sep 16 18:15:52.244327 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:52.244269    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e732fc41-9e6d-4270-ba76-88c22c5df822-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq\" (UID: \"e732fc41-9e6d-4270-ba76-88c22c5df822\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq"
Sep 16 18:15:52.246130 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:52.246103    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e732fc41-9e6d-4270-ba76-88c22c5df822-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq\" (UID: \"e732fc41-9e6d-4270-ba76-88c22c5df822\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq"
Sep 16 18:15:52.246292 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:52.246273    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e732fc41-9e6d-4270-ba76-88c22c5df822-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq\" (UID: \"e732fc41-9e6d-4270-ba76-88c22c5df822\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq"
Sep 16 18:15:52.256526 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:52.256502    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt9xs\" (UniqueName: \"kubernetes.io/projected/e732fc41-9e6d-4270-ba76-88c22c5df822-kube-api-access-rt9xs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq\" (UID: \"e732fc41-9e6d-4270-ba76-88c22c5df822\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq"
Sep 16 18:15:52.262960 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:52.262723    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq"
Sep 16 18:15:52.539720 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:52.539689    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq"]
Sep 16 18:15:52.542330 ip-10-0-136-122 kubenswrapper[2577]: W0916 18:15:52.542304    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode732fc41_9e6d_4270_ba76_88c22c5df822.slice/crio-dc966ebaaa78f95e19aeec523af55529be049b50445836f54ee1b8bb3bcb8f8e WatchSource:0}: Error finding container dc966ebaaa78f95e19aeec523af55529be049b50445836f54ee1b8bb3bcb8f8e: Status 404 returned error can't find the container with id dc966ebaaa78f95e19aeec523af55529be049b50445836f54ee1b8bb3bcb8f8e
Sep 16 18:15:53.275462 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:53.275427    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq" event={"ID":"e732fc41-9e6d-4270-ba76-88c22c5df822","Type":"ContainerStarted","Data":"471a4ca0fc41158a47b6e6de686b5b3fa7ade1e424bbc8192e773aaa70ee176b"}
Sep 16 18:15:53.275818 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:53.275479    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq" event={"ID":"e732fc41-9e6d-4270-ba76-88c22c5df822","Type":"ContainerStarted","Data":"dc966ebaaa78f95e19aeec523af55529be049b50445836f54ee1b8bb3bcb8f8e"}
Sep 16 18:15:53.297887 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:53.297798    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq" podStartSLOduration=2.296851958 podStartE2EDuration="2.296851958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:15:53.293604253 +0000 UTC m=+3074.656283036" watchObservedRunningTime="2026-09-16 18:15:53.296851958 +0000 UTC m=+3074.659530745"
Sep 16 18:15:54.265588 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:54.265542    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq"
Sep 16 18:15:59.441653 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:59.441609    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:15:59.509578 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:15:59.509541    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:01.203640 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:16:01.203151    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg" podUID="07035df6-6670-48b6-bb28-09c19f657aae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": read tcp 10.133.0.2:60776->10.133.0.78:15021: read: connection reset by peer"
Sep 16 18:16:01.331844 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:16:01.331800    2577 generic.go:356] "Generic (PLEG): container finished" podID="07035df6-6670-48b6-bb28-09c19f657aae" containerID="78cd3c84bf4e8170fce6238aa86072a4880c46f7aa6dd3d9c44b5eb30538bb58" exitCode=0
Sep 16 18:16:01.332050 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:16:01.331894    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg" event={"ID":"07035df6-6670-48b6-bb28-09c19f657aae","Type":"ContainerDied","Data":"78cd3c84bf4e8170fce6238aa86072a4880c46f7aa6dd3d9c44b5eb30538bb58"}
Sep 16 18:16:01.332339 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:16:01.332321    2577 scope.go:117] "RemoveContainer" containerID="78cd3c84bf4e8170fce6238aa86072a4880c46f7aa6dd3d9c44b5eb30538bb58"
Sep 16 18:16:01.333324 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:16:01.333286    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:16:01.333381 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:16:01.333366    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:16:01.333431 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:16:01.333419    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:16:02.265694 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:16:02.265651    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq"
Sep 16 18:16:02.278562 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:16:02.278534    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq"
Sep 16 18:16:02.339620 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:16:02.339577    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg" event={"ID":"07035df6-6670-48b6-bb28-09c19f657aae","Type":"ContainerStarted","Data":"019f9384e7332b2848474a7fe260e4723720110cf0784722ef3de440fea26d50"}
Sep 16 18:16:02.350852 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:16:02.350820    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq"
Sep 16 18:16:02.375272 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:16:02.375244    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:16:02.375460 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:16:02.375435    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg" podUID="07035df6-6670-48b6-bb28-09c19f657aae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Sep 16 18:16:03.374743 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:16:03.374704    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg" podUID="07035df6-6670-48b6-bb28-09c19f657aae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Sep 16 18:16:04.375383 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:16:04.375344    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg" podUID="07035df6-6670-48b6-bb28-09c19f657aae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Sep 16 18:16:05.375364 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:16:05.375322    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg" podUID="07035df6-6670-48b6-bb28-09c19f657aae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Sep 16 18:16:06.375453 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:16:06.375416    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg" podUID="07035df6-6670-48b6-bb28-09c19f657aae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Sep 16 18:16:07.378162 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:16:07.378129    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:16:07.378551 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:16:07.378495    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:16:07.379102 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:16:07.379078    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:16:21.905074 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:16:21.905030    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:27.728419 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:16:27.728384    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:27.834743 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:16:27.834703    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:41.160592 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:16:41.160554    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:41.431541 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:16:41.431442    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:16:55.279872 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:16:55.279832    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:14.905136 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:17:14.905095    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:17:15.010667 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:17:15.010629    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:18:04.974819 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:04.974781    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf"]
Sep 16 18:18:04.975443 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:04.975168    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:18:04.984356 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:18:04.984248    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 18:18:04.984356 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:18:04.984356 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:18:04.984356 ip-10-0-136-122 kubenswrapper[2577]:  > containerID="ed94d57fdc7a0ae67252e557afdb17f289f27a905e196201228acc362963b05d" cmd=["/bin/sleep","15"]
Sep 16 18:18:04.984356 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:04.984307    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf" message="" err=<
Sep 16 18:18:04.984356 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:18:04.984356 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:18:04.984356 ip-10-0-136-122 kubenswrapper[2577]:  >
Sep 16 18:18:04.984356 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:18:04.984320    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 18:18:04.984356 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:18:04.984356 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:18:04.984356 ip-10-0-136-122 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf" podUID="9e4dc259-a259-42f3-9fc1-74c4e9d45adc" containerName="main" containerID="cri-o://ed94d57fdc7a0ae67252e557afdb17f289f27a905e196201228acc362963b05d"
Sep 16 18:18:04.985107 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:04.984384    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf" podUID="9e4dc259-a259-42f3-9fc1-74c4e9d45adc" containerName="main" containerID="cri-o://ed94d57fdc7a0ae67252e557afdb17f289f27a905e196201228acc362963b05d" gracePeriod=60
Sep 16 18:18:05.043705 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:05.043669    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:18:05.119420 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:05.119381    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf" podUID="9e4dc259-a259-42f3-9fc1-74c4e9d45adc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 16 18:18:05.340814 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:05.340783    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf"
Sep 16 18:18:05.445477 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:05.445437    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-tls-certs\") pod \"9e4dc259-a259-42f3-9fc1-74c4e9d45adc\" (UID: \"9e4dc259-a259-42f3-9fc1-74c4e9d45adc\") "
Sep 16 18:18:05.445651 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:05.445497    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2jhv\" (UniqueName: \"kubernetes.io/projected/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-kube-api-access-d2jhv\") pod \"9e4dc259-a259-42f3-9fc1-74c4e9d45adc\" (UID: \"9e4dc259-a259-42f3-9fc1-74c4e9d45adc\") "
Sep 16 18:18:05.445651 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:05.445526    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-model-cache\") pod \"9e4dc259-a259-42f3-9fc1-74c4e9d45adc\" (UID: \"9e4dc259-a259-42f3-9fc1-74c4e9d45adc\") "
Sep 16 18:18:05.445651 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:05.445562    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-dshm\") pod \"9e4dc259-a259-42f3-9fc1-74c4e9d45adc\" (UID: \"9e4dc259-a259-42f3-9fc1-74c4e9d45adc\") "
Sep 16 18:18:05.445651 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:05.445639    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-home\") pod \"9e4dc259-a259-42f3-9fc1-74c4e9d45adc\" (UID: \"9e4dc259-a259-42f3-9fc1-74c4e9d45adc\") "
Sep 16 18:18:05.445923 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:05.445677    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-tmp-dir\") pod \"9e4dc259-a259-42f3-9fc1-74c4e9d45adc\" (UID: \"9e4dc259-a259-42f3-9fc1-74c4e9d45adc\") "
Sep 16 18:18:05.445923 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:05.445840    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-model-cache" (OuterVolumeSpecName: "model-cache") pod "9e4dc259-a259-42f3-9fc1-74c4e9d45adc" (UID: "9e4dc259-a259-42f3-9fc1-74c4e9d45adc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:18:05.446047 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:05.445948    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-home" (OuterVolumeSpecName: "home") pod "9e4dc259-a259-42f3-9fc1-74c4e9d45adc" (UID: "9e4dc259-a259-42f3-9fc1-74c4e9d45adc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:18:05.446047 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:05.446014    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:18:05.446140 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:05.446052    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9e4dc259-a259-42f3-9fc1-74c4e9d45adc" (UID: "9e4dc259-a259-42f3-9fc1-74c4e9d45adc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:18:05.447886 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:05.447850    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9e4dc259-a259-42f3-9fc1-74c4e9d45adc" (UID: "9e4dc259-a259-42f3-9fc1-74c4e9d45adc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:18:05.447961 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:05.447847    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-kube-api-access-d2jhv" (OuterVolumeSpecName: "kube-api-access-d2jhv") pod "9e4dc259-a259-42f3-9fc1-74c4e9d45adc" (UID: "9e4dc259-a259-42f3-9fc1-74c4e9d45adc"). InnerVolumeSpecName "kube-api-access-d2jhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:18:05.447961 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:05.447928    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-dshm" (OuterVolumeSpecName: "dshm") pod "9e4dc259-a259-42f3-9fc1-74c4e9d45adc" (UID: "9e4dc259-a259-42f3-9fc1-74c4e9d45adc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:18:05.547119 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:05.547026    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:18:05.547119 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:05.547062    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2jhv\" (UniqueName: \"kubernetes.io/projected/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-kube-api-access-d2jhv\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:18:05.547119 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:05.547072    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:18:05.547119 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:05.547081    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:18:05.547119 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:05.547089    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e4dc259-a259-42f3-9fc1-74c4e9d45adc-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:18:06.007542 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:06.007500    2577 generic.go:356] "Generic (PLEG): container finished" podID="9e4dc259-a259-42f3-9fc1-74c4e9d45adc" containerID="ed94d57fdc7a0ae67252e557afdb17f289f27a905e196201228acc362963b05d" exitCode=0
Sep 16 18:18:06.007987 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:06.007556    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf"
Sep 16 18:18:06.007987 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:06.007553    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf" event={"ID":"9e4dc259-a259-42f3-9fc1-74c4e9d45adc","Type":"ContainerDied","Data":"ed94d57fdc7a0ae67252e557afdb17f289f27a905e196201228acc362963b05d"}
Sep 16 18:18:06.007987 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:06.007615    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf" event={"ID":"9e4dc259-a259-42f3-9fc1-74c4e9d45adc","Type":"ContainerDied","Data":"6e2e7cdbfe889d2895f84f7c8f32f18bd47f558bf78fd8aef57487350135eb5c"}
Sep 16 18:18:06.007987 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:06.007645    2577 scope.go:117] "RemoveContainer" containerID="ed94d57fdc7a0ae67252e557afdb17f289f27a905e196201228acc362963b05d"
Sep 16 18:18:06.019758 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:06.019739    2577 scope.go:117] "RemoveContainer" containerID="ed94d57fdc7a0ae67252e557afdb17f289f27a905e196201228acc362963b05d"
Sep 16 18:18:06.020107 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:18:06.020089    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed94d57fdc7a0ae67252e557afdb17f289f27a905e196201228acc362963b05d\": container with ID starting with ed94d57fdc7a0ae67252e557afdb17f289f27a905e196201228acc362963b05d not found: ID does not exist" containerID="ed94d57fdc7a0ae67252e557afdb17f289f27a905e196201228acc362963b05d"
Sep 16 18:18:06.020164 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:06.020116    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed94d57fdc7a0ae67252e557afdb17f289f27a905e196201228acc362963b05d"} err="failed to get container status \"ed94d57fdc7a0ae67252e557afdb17f289f27a905e196201228acc362963b05d\": rpc error: code = NotFound desc = could not find container \"ed94d57fdc7a0ae67252e557afdb17f289f27a905e196201228acc362963b05d\": container with ID starting with ed94d57fdc7a0ae67252e557afdb17f289f27a905e196201228acc362963b05d not found: ID does not exist"
Sep 16 18:18:06.033520 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:06.033493    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf"]
Sep 16 18:18:06.036753 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:06.036719    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969vpbmf"]
Sep 16 18:18:07.080806 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:07.080769    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e4dc259-a259-42f3-9fc1-74c4e9d45adc" path="/var/lib/kubelet/pods/9e4dc259-a259-42f3-9fc1-74c4e9d45adc/volumes"
Sep 16 18:18:19.257854 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:19.257813    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq"]
Sep 16 18:18:19.266675 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:18:19.266627    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 18:18:19.266675 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:18:19.266675 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:18:19.266675 ip-10-0-136-122 kubenswrapper[2577]:  > containerID="471a4ca0fc41158a47b6e6de686b5b3fa7ade1e424bbc8192e773aaa70ee176b" cmd=["/bin/sleep","15"]
Sep 16 18:18:19.266961 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:19.266686    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq" message="" err=<
Sep 16 18:18:19.266961 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:18:19.266961 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:18:19.266961 ip-10-0-136-122 kubenswrapper[2577]:  >
Sep 16 18:18:19.266961 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:18:19.266698    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 18:18:19.266961 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:18:19.266961 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:18:19.266961 ip-10-0-136-122 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq" podUID="e732fc41-9e6d-4270-ba76-88c22c5df822" containerName="main" containerID="cri-o://471a4ca0fc41158a47b6e6de686b5b3fa7ade1e424bbc8192e773aaa70ee176b"
Sep 16 18:18:19.266961 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:19.266746    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq" podUID="e732fc41-9e6d-4270-ba76-88c22c5df822" containerName="main" containerID="cri-o://471a4ca0fc41158a47b6e6de686b5b3fa7ade1e424bbc8192e773aaa70ee176b" gracePeriod=60
Sep 16 18:18:19.274911 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:19.274880    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:18:19.340975 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:19.340941    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq" podUID="e732fc41-9e6d-4270-ba76-88c22c5df822" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": dial tcp 10.133.0.79:8000: connect: connection refused"
Sep 16 18:18:19.622856 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:19.622824    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq"
Sep 16 18:18:19.773267 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:19.773230    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e732fc41-9e6d-4270-ba76-88c22c5df822-dshm\") pod \"e732fc41-9e6d-4270-ba76-88c22c5df822\" (UID: \"e732fc41-9e6d-4270-ba76-88c22c5df822\") "
Sep 16 18:18:19.773463 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:19.773322    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e732fc41-9e6d-4270-ba76-88c22c5df822-tls-certs\") pod \"e732fc41-9e6d-4270-ba76-88c22c5df822\" (UID: \"e732fc41-9e6d-4270-ba76-88c22c5df822\") "
Sep 16 18:18:19.773463 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:19.773364    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rt9xs\" (UniqueName: \"kubernetes.io/projected/e732fc41-9e6d-4270-ba76-88c22c5df822-kube-api-access-rt9xs\") pod \"e732fc41-9e6d-4270-ba76-88c22c5df822\" (UID: \"e732fc41-9e6d-4270-ba76-88c22c5df822\") "
Sep 16 18:18:19.773463 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:19.773389    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e732fc41-9e6d-4270-ba76-88c22c5df822-model-cache\") pod \"e732fc41-9e6d-4270-ba76-88c22c5df822\" (UID: \"e732fc41-9e6d-4270-ba76-88c22c5df822\") "
Sep 16 18:18:19.773641 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:19.773501    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e732fc41-9e6d-4270-ba76-88c22c5df822-home\") pod \"e732fc41-9e6d-4270-ba76-88c22c5df822\" (UID: \"e732fc41-9e6d-4270-ba76-88c22c5df822\") "
Sep 16 18:18:19.773641 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:19.773537    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e732fc41-9e6d-4270-ba76-88c22c5df822-tmp-dir\") pod \"e732fc41-9e6d-4270-ba76-88c22c5df822\" (UID: \"e732fc41-9e6d-4270-ba76-88c22c5df822\") "
Sep 16 18:18:19.773767 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:19.773656    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e732fc41-9e6d-4270-ba76-88c22c5df822-model-cache" (OuterVolumeSpecName: "model-cache") pod "e732fc41-9e6d-4270-ba76-88c22c5df822" (UID: "e732fc41-9e6d-4270-ba76-88c22c5df822"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:18:19.773767 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:19.773726    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e732fc41-9e6d-4270-ba76-88c22c5df822-home" (OuterVolumeSpecName: "home") pod "e732fc41-9e6d-4270-ba76-88c22c5df822" (UID: "e732fc41-9e6d-4270-ba76-88c22c5df822"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:18:19.773850 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:19.773813    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e732fc41-9e6d-4270-ba76-88c22c5df822-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e732fc41-9e6d-4270-ba76-88c22c5df822" (UID: "e732fc41-9e6d-4270-ba76-88c22c5df822"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:18:19.773932 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:19.773913    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e732fc41-9e6d-4270-ba76-88c22c5df822-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:18:19.773975 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:19.773936    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e732fc41-9e6d-4270-ba76-88c22c5df822-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:18:19.773975 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:19.773950    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e732fc41-9e6d-4270-ba76-88c22c5df822-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:18:19.775542 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:19.775510    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e732fc41-9e6d-4270-ba76-88c22c5df822-dshm" (OuterVolumeSpecName: "dshm") pod "e732fc41-9e6d-4270-ba76-88c22c5df822" (UID: "e732fc41-9e6d-4270-ba76-88c22c5df822"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:18:19.775676 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:19.775605    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e732fc41-9e6d-4270-ba76-88c22c5df822-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e732fc41-9e6d-4270-ba76-88c22c5df822" (UID: "e732fc41-9e6d-4270-ba76-88c22c5df822"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:18:19.775676 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:19.775631    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e732fc41-9e6d-4270-ba76-88c22c5df822-kube-api-access-rt9xs" (OuterVolumeSpecName: "kube-api-access-rt9xs") pod "e732fc41-9e6d-4270-ba76-88c22c5df822" (UID: "e732fc41-9e6d-4270-ba76-88c22c5df822"). InnerVolumeSpecName "kube-api-access-rt9xs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:18:19.875134 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:19.875091    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt9xs\" (UniqueName: \"kubernetes.io/projected/e732fc41-9e6d-4270-ba76-88c22c5df822-kube-api-access-rt9xs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:18:19.875134 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:19.875129    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e732fc41-9e6d-4270-ba76-88c22c5df822-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:18:19.875134 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:19.875139    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e732fc41-9e6d-4270-ba76-88c22c5df822-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:18:20.091077 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:20.091041    2577 generic.go:356] "Generic (PLEG): container finished" podID="e732fc41-9e6d-4270-ba76-88c22c5df822" containerID="471a4ca0fc41158a47b6e6de686b5b3fa7ade1e424bbc8192e773aaa70ee176b" exitCode=0
Sep 16 18:18:20.091244 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:20.091084    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq"
Sep 16 18:18:20.091244 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:20.091128    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq" event={"ID":"e732fc41-9e6d-4270-ba76-88c22c5df822","Type":"ContainerDied","Data":"471a4ca0fc41158a47b6e6de686b5b3fa7ade1e424bbc8192e773aaa70ee176b"}
Sep 16 18:18:20.091244 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:20.091176    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq" event={"ID":"e732fc41-9e6d-4270-ba76-88c22c5df822","Type":"ContainerDied","Data":"dc966ebaaa78f95e19aeec523af55529be049b50445836f54ee1b8bb3bcb8f8e"}
Sep 16 18:18:20.091244 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:20.091202    2577 scope.go:117] "RemoveContainer" containerID="471a4ca0fc41158a47b6e6de686b5b3fa7ade1e424bbc8192e773aaa70ee176b"
Sep 16 18:18:20.100677 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:20.100657    2577 scope.go:117] "RemoveContainer" containerID="471a4ca0fc41158a47b6e6de686b5b3fa7ade1e424bbc8192e773aaa70ee176b"
Sep 16 18:18:20.101038 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:18:20.101018    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"471a4ca0fc41158a47b6e6de686b5b3fa7ade1e424bbc8192e773aaa70ee176b\": container with ID starting with 471a4ca0fc41158a47b6e6de686b5b3fa7ade1e424bbc8192e773aaa70ee176b not found: ID does not exist" containerID="471a4ca0fc41158a47b6e6de686b5b3fa7ade1e424bbc8192e773aaa70ee176b"
Sep 16 18:18:20.101084 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:20.101047    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"471a4ca0fc41158a47b6e6de686b5b3fa7ade1e424bbc8192e773aaa70ee176b"} err="failed to get container status \"471a4ca0fc41158a47b6e6de686b5b3fa7ade1e424bbc8192e773aaa70ee176b\": rpc error: code = NotFound desc = could not find container \"471a4ca0fc41158a47b6e6de686b5b3fa7ade1e424bbc8192e773aaa70ee176b\": container with ID starting with 471a4ca0fc41158a47b6e6de686b5b3fa7ade1e424bbc8192e773aaa70ee176b not found: ID does not exist"
Sep 16 18:18:20.113841 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:20.113816    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq"]
Sep 16 18:18:20.117558 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:20.117540    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ntzrq"]
Sep 16 18:18:21.081524 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:21.081487    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e732fc41-9e6d-4270-ba76-88c22c5df822" path="/var/lib/kubelet/pods/e732fc41-9e6d-4270-ba76-88c22c5df822/volumes"
Sep 16 18:18:24.266549 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:24.266512    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:18:24.362837 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:24.362805    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:18:29.323884 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:29.323822    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"]
Sep 16 18:18:29.324289 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:29.324038    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg" podUID="07035df6-6670-48b6-bb28-09c19f657aae" containerName="istio-proxy" containerID="cri-o://019f9384e7332b2848474a7fe260e4723720110cf0784722ef3de440fea26d50" gracePeriod=30
Sep 16 18:18:29.419777 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:29.419745    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:18:31.253401 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.253319    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"]
Sep 16 18:18:31.253832 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.253737    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e732fc41-9e6d-4270-ba76-88c22c5df822" containerName="main"
Sep 16 18:18:31.253832 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.253752    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e732fc41-9e6d-4270-ba76-88c22c5df822" containerName="main"
Sep 16 18:18:31.253832 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.253764    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e4dc259-a259-42f3-9fc1-74c4e9d45adc" containerName="main"
Sep 16 18:18:31.253832 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.253769    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e4dc259-a259-42f3-9fc1-74c4e9d45adc" containerName="main"
Sep 16 18:18:31.253997 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.253845    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e732fc41-9e6d-4270-ba76-88c22c5df822" containerName="main"
Sep 16 18:18:31.253997 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.253857    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e4dc259-a259-42f3-9fc1-74c4e9d45adc" containerName="main"
Sep 16 18:18:31.256990 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.256972    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"
Sep 16 18:18:31.260061 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.260032    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-rh68r\""
Sep 16 18:18:31.260209 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.260066    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:18:31.260209 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.260109    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Sep 16 18:18:31.260351 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.260330    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 16 18:18:31.260638 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.260623    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 16 18:18:31.260994 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.260980    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 16 18:18:31.282929 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.282896    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"]
Sep 16 18:18:31.367050 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.367010    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8274f2c7-726c-46d6-ac68-f28dfd6382fc-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd\" (UID: \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"
Sep 16 18:18:31.367050 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.367055    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8274f2c7-726c-46d6-ac68-f28dfd6382fc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd\" (UID: \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"
Sep 16 18:18:31.367250 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.367086    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8274f2c7-726c-46d6-ac68-f28dfd6382fc-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd\" (UID: \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"
Sep 16 18:18:31.367250 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.367101    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8274f2c7-726c-46d6-ac68-f28dfd6382fc-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd\" (UID: \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"
Sep 16 18:18:31.367250 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.367206    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f55k\" (UniqueName: \"kubernetes.io/projected/8274f2c7-726c-46d6-ac68-f28dfd6382fc-kube-api-access-8f55k\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd\" (UID: \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"
Sep 16 18:18:31.367250 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.367239    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8274f2c7-726c-46d6-ac68-f28dfd6382fc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd\" (UID: \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"
Sep 16 18:18:31.367385 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.367269    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8274f2c7-726c-46d6-ac68-f28dfd6382fc-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd\" (UID: \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"
Sep 16 18:18:31.367385 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.367304    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8274f2c7-726c-46d6-ac68-f28dfd6382fc-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd\" (UID: \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"
Sep 16 18:18:31.386959 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.386924    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:18:31.468159 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.468123    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f55k\" (UniqueName: \"kubernetes.io/projected/8274f2c7-726c-46d6-ac68-f28dfd6382fc-kube-api-access-8f55k\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd\" (UID: \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"
Sep 16 18:18:31.468159 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.468160    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8274f2c7-726c-46d6-ac68-f28dfd6382fc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd\" (UID: \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"
Sep 16 18:18:31.468373 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.468192    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8274f2c7-726c-46d6-ac68-f28dfd6382fc-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd\" (UID: \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"
Sep 16 18:18:31.468373 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.468218    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8274f2c7-726c-46d6-ac68-f28dfd6382fc-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd\" (UID: \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"
Sep 16 18:18:31.468373 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.468256    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8274f2c7-726c-46d6-ac68-f28dfd6382fc-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd\" (UID: \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"
Sep 16 18:18:31.468547 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.468423    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8274f2c7-726c-46d6-ac68-f28dfd6382fc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd\" (UID: \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"
Sep 16 18:18:31.468547 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.468496    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8274f2c7-726c-46d6-ac68-f28dfd6382fc-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd\" (UID: \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"
Sep 16 18:18:31.468547 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.468527    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8274f2c7-726c-46d6-ac68-f28dfd6382fc-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd\" (UID: \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"
Sep 16 18:18:31.468702 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.468613    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8274f2c7-726c-46d6-ac68-f28dfd6382fc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd\" (UID: \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"
Sep 16 18:18:31.468750 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.468706    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8274f2c7-726c-46d6-ac68-f28dfd6382fc-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd\" (UID: \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"
Sep 16 18:18:31.468910 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.468887    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8274f2c7-726c-46d6-ac68-f28dfd6382fc-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd\" (UID: \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"
Sep 16 18:18:31.468972 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.468897    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8274f2c7-726c-46d6-ac68-f28dfd6382fc-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd\" (UID: \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"
Sep 16 18:18:31.469131 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.469113    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8274f2c7-726c-46d6-ac68-f28dfd6382fc-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd\" (UID: \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"
Sep 16 18:18:31.470599 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.470579    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8274f2c7-726c-46d6-ac68-f28dfd6382fc-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd\" (UID: \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"
Sep 16 18:18:31.471070 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.471047    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8274f2c7-726c-46d6-ac68-f28dfd6382fc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd\" (UID: \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"
Sep 16 18:18:31.476305 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.476284    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f55k\" (UniqueName: \"kubernetes.io/projected/8274f2c7-726c-46d6-ac68-f28dfd6382fc-kube-api-access-8f55k\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd\" (UID: \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"
Sep 16 18:18:31.567647 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.567615    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"
Sep 16 18:18:31.640646 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.640609    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:18:31.853420 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:31.853392    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"]
Sep 16 18:18:31.855299 ip-10-0-136-122 kubenswrapper[2577]: W0916 18:18:31.855268    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8274f2c7_726c_46d6_ac68_f28dfd6382fc.slice/crio-1ea8c8e8fa2e0db7c133530b7a01f32c7b05575fdf595d9d566a59315eaa18c9 WatchSource:0}: Error finding container 1ea8c8e8fa2e0db7c133530b7a01f32c7b05575fdf595d9d566a59315eaa18c9: Status 404 returned error can't find the container with id 1ea8c8e8fa2e0db7c133530b7a01f32c7b05575fdf595d9d566a59315eaa18c9
Sep 16 18:18:32.153018 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:32.152920    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd" event={"ID":"8274f2c7-726c-46d6-ac68-f28dfd6382fc","Type":"ContainerStarted","Data":"161d71c56cb79567d31d4b2044bfc62e1405008f1f3cab56dbcc6d41ff08662b"}
Sep 16 18:18:32.153018 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:32.152976    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd" event={"ID":"8274f2c7-726c-46d6-ac68-f28dfd6382fc","Type":"ContainerStarted","Data":"1ea8c8e8fa2e0db7c133530b7a01f32c7b05575fdf595d9d566a59315eaa18c9"}
Sep 16 18:18:34.577830 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:34.577799    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:18:34.596636 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:34.596611    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07035df6-6670-48b6-bb28-09c19f657aae-workload-socket\") pod \"07035df6-6670-48b6-bb28-09c19f657aae\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") "
Sep 16 18:18:34.596757 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:34.596645    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07035df6-6670-48b6-bb28-09c19f657aae-istio-token\") pod \"07035df6-6670-48b6-bb28-09c19f657aae\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") "
Sep 16 18:18:34.596757 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:34.596695    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcvgm\" (UniqueName: \"kubernetes.io/projected/07035df6-6670-48b6-bb28-09c19f657aae-kube-api-access-vcvgm\") pod \"07035df6-6670-48b6-bb28-09c19f657aae\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") "
Sep 16 18:18:34.596757 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:34.596732    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07035df6-6670-48b6-bb28-09c19f657aae-istio-podinfo\") pod \"07035df6-6670-48b6-bb28-09c19f657aae\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") "
Sep 16 18:18:34.596946 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:34.596800    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07035df6-6670-48b6-bb28-09c19f657aae-istio-envoy\") pod \"07035df6-6670-48b6-bb28-09c19f657aae\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") "
Sep 16 18:18:34.596946 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:34.596824    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07035df6-6670-48b6-bb28-09c19f657aae-credential-socket\") pod \"07035df6-6670-48b6-bb28-09c19f657aae\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") "
Sep 16 18:18:34.596946 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:34.596853    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07035df6-6670-48b6-bb28-09c19f657aae-workload-certs\") pod \"07035df6-6670-48b6-bb28-09c19f657aae\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") "
Sep 16 18:18:34.596946 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:34.596883    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07035df6-6670-48b6-bb28-09c19f657aae-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "07035df6-6670-48b6-bb28-09c19f657aae" (UID: "07035df6-6670-48b6-bb28-09c19f657aae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:18:34.596946 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:34.596890    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07035df6-6670-48b6-bb28-09c19f657aae-istiod-ca-cert\") pod \"07035df6-6670-48b6-bb28-09c19f657aae\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") "
Sep 16 18:18:34.597223 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:34.596961    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07035df6-6670-48b6-bb28-09c19f657aae-istio-data\") pod \"07035df6-6670-48b6-bb28-09c19f657aae\" (UID: \"07035df6-6670-48b6-bb28-09c19f657aae\") "
Sep 16 18:18:34.597279 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:34.597261    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07035df6-6670-48b6-bb28-09c19f657aae-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "07035df6-6670-48b6-bb28-09c19f657aae" (UID: "07035df6-6670-48b6-bb28-09c19f657aae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:18:34.597337 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:34.597277    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07035df6-6670-48b6-bb28-09c19f657aae-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "07035df6-6670-48b6-bb28-09c19f657aae" (UID: "07035df6-6670-48b6-bb28-09c19f657aae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:18:34.597611 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:34.597402    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07035df6-6670-48b6-bb28-09c19f657aae-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "07035df6-6670-48b6-bb28-09c19f657aae" (UID: "07035df6-6670-48b6-bb28-09c19f657aae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:18:34.597611 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:34.597552    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07035df6-6670-48b6-bb28-09c19f657aae-istio-data" (OuterVolumeSpecName: "istio-data") pod "07035df6-6670-48b6-bb28-09c19f657aae" (UID: "07035df6-6670-48b6-bb28-09c19f657aae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:18:34.597913 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:34.597889    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07035df6-6670-48b6-bb28-09c19f657aae-credential-socket\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:18:34.598099 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:34.598074    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07035df6-6670-48b6-bb28-09c19f657aae-workload-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:18:34.598099 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:34.598103    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07035df6-6670-48b6-bb28-09c19f657aae-istiod-ca-cert\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:18:34.598286 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:34.598116    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07035df6-6670-48b6-bb28-09c19f657aae-istio-data\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:18:34.598286 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:34.598129    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07035df6-6670-48b6-bb28-09c19f657aae-workload-socket\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:18:34.599154 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:34.599130    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07035df6-6670-48b6-bb28-09c19f657aae-istio-token" (OuterVolumeSpecName: "istio-token") pod "07035df6-6670-48b6-bb28-09c19f657aae" (UID: "07035df6-6670-48b6-bb28-09c19f657aae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:18:34.599431 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:34.599404    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07035df6-6670-48b6-bb28-09c19f657aae-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "07035df6-6670-48b6-bb28-09c19f657aae" (UID: "07035df6-6670-48b6-bb28-09c19f657aae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:18:34.599510 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:34.599456    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07035df6-6670-48b6-bb28-09c19f657aae-kube-api-access-vcvgm" (OuterVolumeSpecName: "kube-api-access-vcvgm") pod "07035df6-6670-48b6-bb28-09c19f657aae" (UID: "07035df6-6670-48b6-bb28-09c19f657aae"). InnerVolumeSpecName "kube-api-access-vcvgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:18:34.599571 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:34.599549    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07035df6-6670-48b6-bb28-09c19f657aae-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "07035df6-6670-48b6-bb28-09c19f657aae" (UID: "07035df6-6670-48b6-bb28-09c19f657aae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:18:34.698560 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:34.698526    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07035df6-6670-48b6-bb28-09c19f657aae-istio-token\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:18:34.698560 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:34.698555    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcvgm\" (UniqueName: \"kubernetes.io/projected/07035df6-6670-48b6-bb28-09c19f657aae-kube-api-access-vcvgm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:18:34.698730 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:34.698567    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07035df6-6670-48b6-bb28-09c19f657aae-istio-podinfo\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:18:34.698730 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:34.698575    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07035df6-6670-48b6-bb28-09c19f657aae-istio-envoy\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:18:35.172009 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:35.171978    2577 generic.go:356] "Generic (PLEG): container finished" podID="07035df6-6670-48b6-bb28-09c19f657aae" containerID="019f9384e7332b2848474a7fe260e4723720110cf0784722ef3de440fea26d50" exitCode=0
Sep 16 18:18:35.172207 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:35.172048    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"
Sep 16 18:18:35.172207 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:35.172053    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg" event={"ID":"07035df6-6670-48b6-bb28-09c19f657aae","Type":"ContainerDied","Data":"019f9384e7332b2848474a7fe260e4723720110cf0784722ef3de440fea26d50"}
Sep 16 18:18:35.172207 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:35.172174    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg" event={"ID":"07035df6-6670-48b6-bb28-09c19f657aae","Type":"ContainerDied","Data":"b8e0e04fa6a57bab06f2236188365e933a6f0301fa41af798d742625f53eee44"}
Sep 16 18:18:35.172207 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:35.172202    2577 scope.go:117] "RemoveContainer" containerID="019f9384e7332b2848474a7fe260e4723720110cf0784722ef3de440fea26d50"
Sep 16 18:18:35.173811 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:35.173779    2577 generic.go:356] "Generic (PLEG): container finished" podID="8274f2c7-726c-46d6-ac68-f28dfd6382fc" containerID="161d71c56cb79567d31d4b2044bfc62e1405008f1f3cab56dbcc6d41ff08662b" exitCode=0
Sep 16 18:18:35.173934 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:35.173829    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd" event={"ID":"8274f2c7-726c-46d6-ac68-f28dfd6382fc","Type":"ContainerDied","Data":"161d71c56cb79567d31d4b2044bfc62e1405008f1f3cab56dbcc6d41ff08662b"}
Sep 16 18:18:35.181448 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:35.181400    2577 scope.go:117] "RemoveContainer" containerID="78cd3c84bf4e8170fce6238aa86072a4880c46f7aa6dd3d9c44b5eb30538bb58"
Sep 16 18:18:35.189190 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:35.189163    2577 scope.go:117] "RemoveContainer" containerID="019f9384e7332b2848474a7fe260e4723720110cf0784722ef3de440fea26d50"
Sep 16 18:18:35.189464 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:18:35.189445    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"019f9384e7332b2848474a7fe260e4723720110cf0784722ef3de440fea26d50\": container with ID starting with 019f9384e7332b2848474a7fe260e4723720110cf0784722ef3de440fea26d50 not found: ID does not exist" containerID="019f9384e7332b2848474a7fe260e4723720110cf0784722ef3de440fea26d50"
Sep 16 18:18:35.189518 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:35.189475    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"019f9384e7332b2848474a7fe260e4723720110cf0784722ef3de440fea26d50"} err="failed to get container status \"019f9384e7332b2848474a7fe260e4723720110cf0784722ef3de440fea26d50\": rpc error: code = NotFound desc = could not find container \"019f9384e7332b2848474a7fe260e4723720110cf0784722ef3de440fea26d50\": container with ID starting with 019f9384e7332b2848474a7fe260e4723720110cf0784722ef3de440fea26d50 not found: ID does not exist"
Sep 16 18:18:35.189518 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:35.189497    2577 scope.go:117] "RemoveContainer" containerID="78cd3c84bf4e8170fce6238aa86072a4880c46f7aa6dd3d9c44b5eb30538bb58"
Sep 16 18:18:35.189797 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:18:35.189775    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78cd3c84bf4e8170fce6238aa86072a4880c46f7aa6dd3d9c44b5eb30538bb58\": container with ID starting with 78cd3c84bf4e8170fce6238aa86072a4880c46f7aa6dd3d9c44b5eb30538bb58 not found: ID does not exist" containerID="78cd3c84bf4e8170fce6238aa86072a4880c46f7aa6dd3d9c44b5eb30538bb58"
Sep 16 18:18:35.189918 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:35.189803    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78cd3c84bf4e8170fce6238aa86072a4880c46f7aa6dd3d9c44b5eb30538bb58"} err="failed to get container status \"78cd3c84bf4e8170fce6238aa86072a4880c46f7aa6dd3d9c44b5eb30538bb58\": rpc error: code = NotFound desc = could not find container \"78cd3c84bf4e8170fce6238aa86072a4880c46f7aa6dd3d9c44b5eb30538bb58\": container with ID starting with 78cd3c84bf4e8170fce6238aa86072a4880c46f7aa6dd3d9c44b5eb30538bb58 not found: ID does not exist"
Sep 16 18:18:35.193856 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:35.193830    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"]
Sep 16 18:18:35.198499 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:35.198477    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-s48rg"]
Sep 16 18:18:36.185785 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:36.185747    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd" event={"ID":"8274f2c7-726c-46d6-ac68-f28dfd6382fc","Type":"ContainerStarted","Data":"d6f73f5a505436783bd3653e217d6eed55444309c7f94ff1b9f403006a9a673d"}
Sep 16 18:18:36.212244 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:36.212185    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd" podStartSLOduration=2.909227422 podStartE2EDuration="5.212168269s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.302940847s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:18:36.209756552 +0000 UTC m=+3237.572435340" watchObservedRunningTime="2026-09-16 18:18:36.212168269 +0000 UTC m=+3237.574847117"
Sep 16 18:18:36.568373 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:36.568340    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"
Sep 16 18:18:37.080631 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:37.080600    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07035df6-6670-48b6-bb28-09c19f657aae" path="/var/lib/kubelet/pods/07035df6-6670-48b6-bb28-09c19f657aae/volumes"
Sep 16 18:18:41.568363 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:41.568318    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"
Sep 16 18:18:41.570317 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:41.570286    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd" podUID="8274f2c7-726c-46d6-ac68-f28dfd6382fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 18:18:41.698951 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:41.698904    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:18:51.069090 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:51.069050    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:18:51.568921 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:51.568885    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd" podUID="8274f2c7-726c-46d6-ac68-f28dfd6382fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 18:18:51.931908 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:51.931802    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:18:55.045569 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:55.045529    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:18:55.159118 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:55.159080    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:18:57.926490 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:57.926444    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:18:59.682479 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:18:59.682447    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:19:01.568098 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:19:01.568054    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd" podUID="8274f2c7-726c-46d6-ac68-f28dfd6382fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 18:19:01.826898 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:19:01.826780    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:19:07.605537 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:19:07.605500    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:19:07.677034 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:19:07.677001    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:19:11.569152 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:19:11.569101    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd" podUID="8274f2c7-726c-46d6-ac68-f28dfd6382fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 18:19:12.476286 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:19:12.476246    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:19:21.151253 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:19:21.151211    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:19:21.568596 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:19:21.568559    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd" podUID="8274f2c7-726c-46d6-ac68-f28dfd6382fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 18:19:21.763506 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:19:21.763460    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:19:21.870041 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:19:21.869928    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:19:31.568377 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:19:31.568329    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd" podUID="8274f2c7-726c-46d6-ac68-f28dfd6382fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 18:19:33.506078 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:19:33.506034    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:19:41.568748 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:19:41.568703    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd" podUID="8274f2c7-726c-46d6-ac68-f28dfd6382fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 18:19:47.416126 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:19:47.416080    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:19:51.568562 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:19:51.568518    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd" podUID="8274f2c7-726c-46d6-ac68-f28dfd6382fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 18:20:01.568371 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:01.568263    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd" podUID="8274f2c7-726c-46d6-ac68-f28dfd6382fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 18:20:11.569060 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:11.569018    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd" podUID="8274f2c7-726c-46d6-ac68-f28dfd6382fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 18:20:21.568349 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:21.568287    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd" podUID="8274f2c7-726c-46d6-ac68-f28dfd6382fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 16 18:20:23.714283 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:23.714232    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:20:31.579142 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:31.579102    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"
Sep 16 18:20:31.589085 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:31.589047    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"
Sep 16 18:20:36.561743 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:36.561701    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:20:36.562226 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:36.561972    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"]
Sep 16 18:20:36.659654 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:36.659614    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:20:42.103138 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:42.103096    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:20:42.200299 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:42.200267    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:20:51.584968 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:51.584910    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd" podUID="8274f2c7-726c-46d6-ac68-f28dfd6382fc" containerName="main" containerID="cri-o://d6f73f5a505436783bd3653e217d6eed55444309c7f94ff1b9f403006a9a673d" gracePeriod=45
Sep 16 18:20:52.340853 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.340826    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"
Sep 16 18:20:52.390887 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.390791    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8274f2c7-726c-46d6-ac68-f28dfd6382fc-home\") pod \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\" (UID: \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\") "
Sep 16 18:20:52.390887 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.390825    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8274f2c7-726c-46d6-ac68-f28dfd6382fc-model-cache\") pod \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\" (UID: \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\") "
Sep 16 18:20:52.390887 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.390845    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8f55k\" (UniqueName: \"kubernetes.io/projected/8274f2c7-726c-46d6-ac68-f28dfd6382fc-kube-api-access-8f55k\") pod \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\" (UID: \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\") "
Sep 16 18:20:52.391168 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.390899    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8274f2c7-726c-46d6-ac68-f28dfd6382fc-tmp-dir\") pod \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\" (UID: \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\") "
Sep 16 18:20:52.391168 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.390954    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8274f2c7-726c-46d6-ac68-f28dfd6382fc-tls-certs\") pod \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\" (UID: \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\") "
Sep 16 18:20:52.391168 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.391015    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8274f2c7-726c-46d6-ac68-f28dfd6382fc-dshm\") pod \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\" (UID: \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\") "
Sep 16 18:20:52.391168 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.391039    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8274f2c7-726c-46d6-ac68-f28dfd6382fc-kserve-provision-location\") pod \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\" (UID: \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\") "
Sep 16 18:20:52.391168 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.391065    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8274f2c7-726c-46d6-ac68-f28dfd6382fc-cabundle-cert\") pod \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\" (UID: \"8274f2c7-726c-46d6-ac68-f28dfd6382fc\") "
Sep 16 18:20:52.391168 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.391128    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8274f2c7-726c-46d6-ac68-f28dfd6382fc-model-cache" (OuterVolumeSpecName: "model-cache") pod "8274f2c7-726c-46d6-ac68-f28dfd6382fc" (UID: "8274f2c7-726c-46d6-ac68-f28dfd6382fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:20:52.391469 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.391358    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8274f2c7-726c-46d6-ac68-f28dfd6382fc-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:20:52.391535 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.391468    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8274f2c7-726c-46d6-ac68-f28dfd6382fc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8274f2c7-726c-46d6-ac68-f28dfd6382fc" (UID: "8274f2c7-726c-46d6-ac68-f28dfd6382fc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:20:52.391665 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.391636    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8274f2c7-726c-46d6-ac68-f28dfd6382fc-home" (OuterVolumeSpecName: "home") pod "8274f2c7-726c-46d6-ac68-f28dfd6382fc" (UID: "8274f2c7-726c-46d6-ac68-f28dfd6382fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:20:52.393500 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.393466    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8274f2c7-726c-46d6-ac68-f28dfd6382fc-dshm" (OuterVolumeSpecName: "dshm") pod "8274f2c7-726c-46d6-ac68-f28dfd6382fc" (UID: "8274f2c7-726c-46d6-ac68-f28dfd6382fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:20:52.393667 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.393638    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8274f2c7-726c-46d6-ac68-f28dfd6382fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8274f2c7-726c-46d6-ac68-f28dfd6382fc" (UID: "8274f2c7-726c-46d6-ac68-f28dfd6382fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:20:52.393733 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.393666    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8274f2c7-726c-46d6-ac68-f28dfd6382fc-kube-api-access-8f55k" (OuterVolumeSpecName: "kube-api-access-8f55k") pod "8274f2c7-726c-46d6-ac68-f28dfd6382fc" (UID: "8274f2c7-726c-46d6-ac68-f28dfd6382fc"). InnerVolumeSpecName "kube-api-access-8f55k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:20:52.405132 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.405105    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8274f2c7-726c-46d6-ac68-f28dfd6382fc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8274f2c7-726c-46d6-ac68-f28dfd6382fc" (UID: "8274f2c7-726c-46d6-ac68-f28dfd6382fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:20:52.407951 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.407924    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8274f2c7-726c-46d6-ac68-f28dfd6382fc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8274f2c7-726c-46d6-ac68-f28dfd6382fc" (UID: "8274f2c7-726c-46d6-ac68-f28dfd6382fc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:20:52.491951 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.491916    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8274f2c7-726c-46d6-ac68-f28dfd6382fc-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:20:52.491951 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.491947    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f55k\" (UniqueName: \"kubernetes.io/projected/8274f2c7-726c-46d6-ac68-f28dfd6382fc-kube-api-access-8f55k\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:20:52.492221 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.491987    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8274f2c7-726c-46d6-ac68-f28dfd6382fc-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:20:52.492221 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.492000    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8274f2c7-726c-46d6-ac68-f28dfd6382fc-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:20:52.492221 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.492011    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8274f2c7-726c-46d6-ac68-f28dfd6382fc-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:20:52.492221 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.492023    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8274f2c7-726c-46d6-ac68-f28dfd6382fc-kserve-provision-location\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:20:52.492221 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.492035    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8274f2c7-726c-46d6-ac68-f28dfd6382fc-cabundle-cert\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:20:52.967117 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.967062    2577 generic.go:356] "Generic (PLEG): container finished" podID="8274f2c7-726c-46d6-ac68-f28dfd6382fc" containerID="d6f73f5a505436783bd3653e217d6eed55444309c7f94ff1b9f403006a9a673d" exitCode=0
Sep 16 18:20:52.967587 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.967154    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"
Sep 16 18:20:52.967587 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.967145    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd" event={"ID":"8274f2c7-726c-46d6-ac68-f28dfd6382fc","Type":"ContainerDied","Data":"d6f73f5a505436783bd3653e217d6eed55444309c7f94ff1b9f403006a9a673d"}
Sep 16 18:20:52.967587 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.967344    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd" event={"ID":"8274f2c7-726c-46d6-ac68-f28dfd6382fc","Type":"ContainerDied","Data":"1ea8c8e8fa2e0db7c133530b7a01f32c7b05575fdf595d9d566a59315eaa18c9"}
Sep 16 18:20:52.967587 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.967387    2577 scope.go:117] "RemoveContainer" containerID="d6f73f5a505436783bd3653e217d6eed55444309c7f94ff1b9f403006a9a673d"
Sep 16 18:20:52.976809 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.976793    2577 scope.go:117] "RemoveContainer" containerID="161d71c56cb79567d31d4b2044bfc62e1405008f1f3cab56dbcc6d41ff08662b"
Sep 16 18:20:52.984525 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.984506    2577 scope.go:117] "RemoveContainer" containerID="d6f73f5a505436783bd3653e217d6eed55444309c7f94ff1b9f403006a9a673d"
Sep 16 18:20:52.984769 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:20:52.984749    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6f73f5a505436783bd3653e217d6eed55444309c7f94ff1b9f403006a9a673d\": container with ID starting with d6f73f5a505436783bd3653e217d6eed55444309c7f94ff1b9f403006a9a673d not found: ID does not exist" containerID="d6f73f5a505436783bd3653e217d6eed55444309c7f94ff1b9f403006a9a673d"
Sep 16 18:20:52.984818 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.984776    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6f73f5a505436783bd3653e217d6eed55444309c7f94ff1b9f403006a9a673d"} err="failed to get container status \"d6f73f5a505436783bd3653e217d6eed55444309c7f94ff1b9f403006a9a673d\": rpc error: code = NotFound desc = could not find container \"d6f73f5a505436783bd3653e217d6eed55444309c7f94ff1b9f403006a9a673d\": container with ID starting with d6f73f5a505436783bd3653e217d6eed55444309c7f94ff1b9f403006a9a673d not found: ID does not exist"
Sep 16 18:20:52.984818 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.984793    2577 scope.go:117] "RemoveContainer" containerID="161d71c56cb79567d31d4b2044bfc62e1405008f1f3cab56dbcc6d41ff08662b"
Sep 16 18:20:52.985083 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:20:52.985060    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"161d71c56cb79567d31d4b2044bfc62e1405008f1f3cab56dbcc6d41ff08662b\": container with ID starting with 161d71c56cb79567d31d4b2044bfc62e1405008f1f3cab56dbcc6d41ff08662b not found: ID does not exist" containerID="161d71c56cb79567d31d4b2044bfc62e1405008f1f3cab56dbcc6d41ff08662b"
Sep 16 18:20:52.985175 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.985087    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"161d71c56cb79567d31d4b2044bfc62e1405008f1f3cab56dbcc6d41ff08662b"} err="failed to get container status \"161d71c56cb79567d31d4b2044bfc62e1405008f1f3cab56dbcc6d41ff08662b\": rpc error: code = NotFound desc = could not find container \"161d71c56cb79567d31d4b2044bfc62e1405008f1f3cab56dbcc6d41ff08662b\": container with ID starting with 161d71c56cb79567d31d4b2044bfc62e1405008f1f3cab56dbcc6d41ff08662b not found: ID does not exist"
Sep 16 18:20:52.992756 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.992734    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"]
Sep 16 18:20:52.996115 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:52.996095    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-696b64d9f6s7jgd"]
Sep 16 18:20:53.080552 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:20:53.080522    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8274f2c7-726c-46d6-ac68-f28dfd6382fc" path="/var/lib/kubelet/pods/8274f2c7-726c-46d6-ac68-f28dfd6382fc/volumes"
Sep 16 18:21:28.048112 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:28.048062    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:21:28.148281 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:28.148241    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:21:31.874086 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:31.873990    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"]
Sep 16 18:21:31.874571 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:31.874549    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8274f2c7-726c-46d6-ac68-f28dfd6382fc" containerName="storage-initializer"
Sep 16 18:21:31.874663 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:31.874573    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8274f2c7-726c-46d6-ac68-f28dfd6382fc" containerName="storage-initializer"
Sep 16 18:21:31.874663 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:31.874585    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07035df6-6670-48b6-bb28-09c19f657aae" containerName="istio-proxy"
Sep 16 18:21:31.874663 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:31.874593    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="07035df6-6670-48b6-bb28-09c19f657aae" containerName="istio-proxy"
Sep 16 18:21:31.874663 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:31.874603    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8274f2c7-726c-46d6-ac68-f28dfd6382fc" containerName="main"
Sep 16 18:21:31.874663 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:31.874611    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8274f2c7-726c-46d6-ac68-f28dfd6382fc" containerName="main"
Sep 16 18:21:31.874663 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:31.874626    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07035df6-6670-48b6-bb28-09c19f657aae" containerName="istio-proxy"
Sep 16 18:21:31.874663 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:31.874635    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="07035df6-6670-48b6-bb28-09c19f657aae" containerName="istio-proxy"
Sep 16 18:21:31.875040 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:31.874757    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="07035df6-6670-48b6-bb28-09c19f657aae" containerName="istio-proxy"
Sep 16 18:21:31.875040 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:31.874770    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8274f2c7-726c-46d6-ac68-f28dfd6382fc" containerName="main"
Sep 16 18:21:31.875040 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:31.874965    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="07035df6-6670-48b6-bb28-09c19f657aae" containerName="istio-proxy"
Sep 16 18:21:31.878222 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:31.878199    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"
Sep 16 18:21:31.883528 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:31.883499    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 16 18:21:31.883693 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:31.883499    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 16 18:21:31.883693 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:31.883535    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 16 18:21:31.885247 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:31.885230    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-2jbxr\""
Sep 16 18:21:31.902437 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:31.902408    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"]
Sep 16 18:21:31.914180 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:31.914157    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2216e73f-5855-4c61-80b3-98a47d81f3e8-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb\" (UID: \"2216e73f-5855-4c61-80b3-98a47d81f3e8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"
Sep 16 18:21:31.914289 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:31.914191    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2216e73f-5855-4c61-80b3-98a47d81f3e8-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb\" (UID: \"2216e73f-5855-4c61-80b3-98a47d81f3e8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"
Sep 16 18:21:31.914289 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:31.914222    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2216e73f-5855-4c61-80b3-98a47d81f3e8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb\" (UID: \"2216e73f-5855-4c61-80b3-98a47d81f3e8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"
Sep 16 18:21:31.914289 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:31.914250    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2216e73f-5855-4c61-80b3-98a47d81f3e8-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb\" (UID: \"2216e73f-5855-4c61-80b3-98a47d81f3e8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"
Sep 16 18:21:31.914433 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:31.914293    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q9jp\" (UniqueName: \"kubernetes.io/projected/2216e73f-5855-4c61-80b3-98a47d81f3e8-kube-api-access-7q9jp\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb\" (UID: \"2216e73f-5855-4c61-80b3-98a47d81f3e8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"
Sep 16 18:21:31.914433 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:31.914350    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2216e73f-5855-4c61-80b3-98a47d81f3e8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb\" (UID: \"2216e73f-5855-4c61-80b3-98a47d81f3e8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"
Sep 16 18:21:31.914433 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:31.914405    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2216e73f-5855-4c61-80b3-98a47d81f3e8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb\" (UID: \"2216e73f-5855-4c61-80b3-98a47d81f3e8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"
Sep 16 18:21:31.951530 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:31.951492    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:21:32.014841 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:32.014799    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q9jp\" (UniqueName: \"kubernetes.io/projected/2216e73f-5855-4c61-80b3-98a47d81f3e8-kube-api-access-7q9jp\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb\" (UID: \"2216e73f-5855-4c61-80b3-98a47d81f3e8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"
Sep 16 18:21:32.015038 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:32.014850    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2216e73f-5855-4c61-80b3-98a47d81f3e8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb\" (UID: \"2216e73f-5855-4c61-80b3-98a47d81f3e8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"
Sep 16 18:21:32.015038 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:32.014917    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2216e73f-5855-4c61-80b3-98a47d81f3e8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb\" (UID: \"2216e73f-5855-4c61-80b3-98a47d81f3e8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"
Sep 16 18:21:32.015038 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:32.014953    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2216e73f-5855-4c61-80b3-98a47d81f3e8-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb\" (UID: \"2216e73f-5855-4c61-80b3-98a47d81f3e8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"
Sep 16 18:21:32.015229 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:32.015075    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2216e73f-5855-4c61-80b3-98a47d81f3e8-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb\" (UID: \"2216e73f-5855-4c61-80b3-98a47d81f3e8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"
Sep 16 18:21:32.015229 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:32.015134    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2216e73f-5855-4c61-80b3-98a47d81f3e8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb\" (UID: \"2216e73f-5855-4c61-80b3-98a47d81f3e8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"
Sep 16 18:21:32.015229 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:32.015158    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2216e73f-5855-4c61-80b3-98a47d81f3e8-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb\" (UID: \"2216e73f-5855-4c61-80b3-98a47d81f3e8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"
Sep 16 18:21:32.015389 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:32.015350    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2216e73f-5855-4c61-80b3-98a47d81f3e8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb\" (UID: \"2216e73f-5855-4c61-80b3-98a47d81f3e8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"
Sep 16 18:21:32.015451 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:32.015413    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2216e73f-5855-4c61-80b3-98a47d81f3e8-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb\" (UID: \"2216e73f-5855-4c61-80b3-98a47d81f3e8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"
Sep 16 18:21:32.015509 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:32.015455    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2216e73f-5855-4c61-80b3-98a47d81f3e8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb\" (UID: \"2216e73f-5855-4c61-80b3-98a47d81f3e8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"
Sep 16 18:21:32.015552 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:32.015529    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2216e73f-5855-4c61-80b3-98a47d81f3e8-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb\" (UID: \"2216e73f-5855-4c61-80b3-98a47d81f3e8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"
Sep 16 18:21:32.017433 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:32.017413    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2216e73f-5855-4c61-80b3-98a47d81f3e8-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb\" (UID: \"2216e73f-5855-4c61-80b3-98a47d81f3e8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"
Sep 16 18:21:32.017619 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:32.017603    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2216e73f-5855-4c61-80b3-98a47d81f3e8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb\" (UID: \"2216e73f-5855-4c61-80b3-98a47d81f3e8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"
Sep 16 18:21:32.025666 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:32.025641    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q9jp\" (UniqueName: \"kubernetes.io/projected/2216e73f-5855-4c61-80b3-98a47d81f3e8-kube-api-access-7q9jp\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb\" (UID: \"2216e73f-5855-4c61-80b3-98a47d81f3e8\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"
Sep 16 18:21:32.137626 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:32.137542    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:21:32.188195 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:32.188159    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"
Sep 16 18:21:32.229661 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:32.229616    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:21:32.495482 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:32.495455    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"]
Sep 16 18:21:32.534006 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:32.533985    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:21:33.017670 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:33.017635    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:21:33.174217 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:33.174181    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb" event={"ID":"2216e73f-5855-4c61-80b3-98a47d81f3e8","Type":"ContainerStarted","Data":"ac155eac1631dd590e62dc2f82c1d22ce0a3003e7bd4c266d95ff316aeb91aca"}
Sep 16 18:21:35.234581 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:35.234536    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:21:38.533417 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:38.533373    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:21:41.218100 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:41.218057    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:21:41.318496 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:41.318461    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:21:42.229322 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:42.229284    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb" event={"ID":"2216e73f-5855-4c61-80b3-98a47d81f3e8","Type":"ContainerStarted","Data":"bcabfaa52ad87775924f91d8b909d73b2feafe8081420c3d662ed4ce68d97a96"}
Sep 16 18:21:44.240425 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:44.240390    2577 generic.go:356] "Generic (PLEG): container finished" podID="2216e73f-5855-4c61-80b3-98a47d81f3e8" containerID="bcabfaa52ad87775924f91d8b909d73b2feafe8081420c3d662ed4ce68d97a96" exitCode=0
Sep 16 18:21:44.240941 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:44.240467    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb" event={"ID":"2216e73f-5855-4c61-80b3-98a47d81f3e8","Type":"ContainerDied","Data":"bcabfaa52ad87775924f91d8b909d73b2feafe8081420c3d662ed4ce68d97a96"}
Sep 16 18:21:44.709617 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:44.709564    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:21:44.822519 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:44.822400    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:21:45.249096 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:45.249008    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb" event={"ID":"2216e73f-5855-4c61-80b3-98a47d81f3e8","Type":"ContainerStarted","Data":"19a7fa1ebd74a095c35ebb56c065f0feb44fe4b1317abc75ea894f4d1873e1a7"}
Sep 16 18:21:45.249096 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:45.249076    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb" event={"ID":"2216e73f-5855-4c61-80b3-98a47d81f3e8","Type":"ContainerStarted","Data":"246f6c2daaa132bc07f773753b465799d45668049a290ec33f4d856a5cf00828"}
Sep 16 18:21:45.280624 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:45.280568    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb" podStartSLOduration=3.609960707 podStartE2EDuration="14.280551789s" totalImagesPullingTime="9.485071264s" totalInitContainerRuntime="1.185519818s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:21:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:21:45.279193031 +0000 UTC m=+3426.641871818" watchObservedRunningTime="2026-09-16 18:21:45.280551789 +0000 UTC m=+3426.643230600"
Sep 16 18:21:46.189070 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:46.189026    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"
Sep 16 18:21:52.188761 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:52.188714    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"
Sep 16 18:21:52.190664 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:52.190629    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb" podUID="2216e73f-5855-4c61-80b3-98a47d81f3e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": dial tcp 10.133.0.81:8000: connect: connection refused"
Sep 16 18:21:52.807421 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:21:52.807372    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:22:02.188946 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:02.188893    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb" podUID="2216e73f-5855-4c61-80b3-98a47d81f3e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": dial tcp 10.133.0.81:8000: connect: connection refused"
Sep 16 18:22:03.776038 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:03.775996    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"]
Sep 16 18:22:03.781742 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:03.781723    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"
Sep 16 18:22:03.785790 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:03.785763    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 16 18:22:03.785946 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:03.785816    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Sep 16 18:22:03.785946 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:03.785819    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-nv9rh\""
Sep 16 18:22:03.786079 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:03.785820    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:22:03.786256 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:03.786237    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 16 18:22:03.786328 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:03.786310    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 16 18:22:03.792332 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:03.792306    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"]
Sep 16 18:22:03.863881 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:03.863825    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:22:03.900435 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:03.900398    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6zqd\" (UniqueName: \"kubernetes.io/projected/d91c0691-fedf-44a9-9a50-408c6635eca0-kube-api-access-p6zqd\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q\" (UID: \"d91c0691-fedf-44a9-9a50-408c6635eca0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"
Sep 16 18:22:03.900698 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:03.900682    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d91c0691-fedf-44a9-9a50-408c6635eca0-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q\" (UID: \"d91c0691-fedf-44a9-9a50-408c6635eca0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"
Sep 16 18:22:03.900816 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:03.900804    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d91c0691-fedf-44a9-9a50-408c6635eca0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q\" (UID: \"d91c0691-fedf-44a9-9a50-408c6635eca0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"
Sep 16 18:22:03.900947 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:03.900931    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d91c0691-fedf-44a9-9a50-408c6635eca0-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q\" (UID: \"d91c0691-fedf-44a9-9a50-408c6635eca0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"
Sep 16 18:22:03.901058 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:03.901046    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d91c0691-fedf-44a9-9a50-408c6635eca0-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q\" (UID: \"d91c0691-fedf-44a9-9a50-408c6635eca0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"
Sep 16 18:22:03.901157 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:03.901145    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d91c0691-fedf-44a9-9a50-408c6635eca0-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q\" (UID: \"d91c0691-fedf-44a9-9a50-408c6635eca0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"
Sep 16 18:22:03.901278 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:03.901266    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d91c0691-fedf-44a9-9a50-408c6635eca0-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q\" (UID: \"d91c0691-fedf-44a9-9a50-408c6635eca0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"
Sep 16 18:22:03.901369 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:03.901356    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d91c0691-fedf-44a9-9a50-408c6635eca0-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q\" (UID: \"d91c0691-fedf-44a9-9a50-408c6635eca0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"
Sep 16 18:22:04.002136 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:04.002097    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d91c0691-fedf-44a9-9a50-408c6635eca0-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q\" (UID: \"d91c0691-fedf-44a9-9a50-408c6635eca0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"
Sep 16 18:22:04.002136 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:04.002136    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d91c0691-fedf-44a9-9a50-408c6635eca0-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q\" (UID: \"d91c0691-fedf-44a9-9a50-408c6635eca0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"
Sep 16 18:22:04.002358 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:04.002168    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d91c0691-fedf-44a9-9a50-408c6635eca0-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q\" (UID: \"d91c0691-fedf-44a9-9a50-408c6635eca0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"
Sep 16 18:22:04.002358 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:04.002287    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d91c0691-fedf-44a9-9a50-408c6635eca0-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q\" (UID: \"d91c0691-fedf-44a9-9a50-408c6635eca0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"
Sep 16 18:22:04.002449 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:04.002393    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6zqd\" (UniqueName: \"kubernetes.io/projected/d91c0691-fedf-44a9-9a50-408c6635eca0-kube-api-access-p6zqd\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q\" (UID: \"d91c0691-fedf-44a9-9a50-408c6635eca0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"
Sep 16 18:22:04.002449 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:04.002439    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d91c0691-fedf-44a9-9a50-408c6635eca0-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q\" (UID: \"d91c0691-fedf-44a9-9a50-408c6635eca0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"
Sep 16 18:22:04.002556 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:04.002479    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d91c0691-fedf-44a9-9a50-408c6635eca0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q\" (UID: \"d91c0691-fedf-44a9-9a50-408c6635eca0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"
Sep 16 18:22:04.002556 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:04.002515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d91c0691-fedf-44a9-9a50-408c6635eca0-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q\" (UID: \"d91c0691-fedf-44a9-9a50-408c6635eca0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"
Sep 16 18:22:04.002556 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:04.002537    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d91c0691-fedf-44a9-9a50-408c6635eca0-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q\" (UID: \"d91c0691-fedf-44a9-9a50-408c6635eca0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"
Sep 16 18:22:04.002830 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:04.002802    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d91c0691-fedf-44a9-9a50-408c6635eca0-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q\" (UID: \"d91c0691-fedf-44a9-9a50-408c6635eca0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"
Sep 16 18:22:04.002957 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:04.002825    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d91c0691-fedf-44a9-9a50-408c6635eca0-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q\" (UID: \"d91c0691-fedf-44a9-9a50-408c6635eca0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"
Sep 16 18:22:04.003008 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:04.002946    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d91c0691-fedf-44a9-9a50-408c6635eca0-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q\" (UID: \"d91c0691-fedf-44a9-9a50-408c6635eca0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"
Sep 16 18:22:04.003045 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:04.003012    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d91c0691-fedf-44a9-9a50-408c6635eca0-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q\" (UID: \"d91c0691-fedf-44a9-9a50-408c6635eca0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"
Sep 16 18:22:04.004901 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:04.004853    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d91c0691-fedf-44a9-9a50-408c6635eca0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q\" (UID: \"d91c0691-fedf-44a9-9a50-408c6635eca0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"
Sep 16 18:22:04.005020 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:04.004909    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d91c0691-fedf-44a9-9a50-408c6635eca0-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q\" (UID: \"d91c0691-fedf-44a9-9a50-408c6635eca0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"
Sep 16 18:22:04.011456 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:04.011429    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6zqd\" (UniqueName: \"kubernetes.io/projected/d91c0691-fedf-44a9-9a50-408c6635eca0-kube-api-access-p6zqd\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q\" (UID: \"d91c0691-fedf-44a9-9a50-408c6635eca0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"
Sep 16 18:22:04.094281 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:04.094236    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"
Sep 16 18:22:04.220478 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:04.220442    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:22:04.383220 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:04.383184    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"]
Sep 16 18:22:04.384686 ip-10-0-136-122 kubenswrapper[2577]: W0916 18:22:04.384650    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd91c0691_fedf_44a9_9a50_408c6635eca0.slice/crio-5c23f7a29489a3609f6821fb80f85afb9df0dcbd72c708f5a75b33a5cfe16668 WatchSource:0}: Error finding container 5c23f7a29489a3609f6821fb80f85afb9df0dcbd72c708f5a75b33a5cfe16668: Status 404 returned error can't find the container with id 5c23f7a29489a3609f6821fb80f85afb9df0dcbd72c708f5a75b33a5cfe16668
Sep 16 18:22:05.363985 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:05.363942    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q" event={"ID":"d91c0691-fedf-44a9-9a50-408c6635eca0","Type":"ContainerStarted","Data":"45a6ae7e0df370afc69d95e90be7996f04a34446211f3b7207fb32b2bc2a0cc8"}
Sep 16 18:22:05.364370 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:05.364009    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q" event={"ID":"d91c0691-fedf-44a9-9a50-408c6635eca0","Type":"ContainerStarted","Data":"5c23f7a29489a3609f6821fb80f85afb9df0dcbd72c708f5a75b33a5cfe16668"}
Sep 16 18:22:08.382074 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:08.382036    2577 generic.go:356] "Generic (PLEG): container finished" podID="d91c0691-fedf-44a9-9a50-408c6635eca0" containerID="45a6ae7e0df370afc69d95e90be7996f04a34446211f3b7207fb32b2bc2a0cc8" exitCode=0
Sep 16 18:22:08.382518 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:08.382113    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q" event={"ID":"d91c0691-fedf-44a9-9a50-408c6635eca0","Type":"ContainerDied","Data":"45a6ae7e0df370afc69d95e90be7996f04a34446211f3b7207fb32b2bc2a0cc8"}
Sep 16 18:22:09.391923 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:09.391887    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q" event={"ID":"d91c0691-fedf-44a9-9a50-408c6635eca0","Type":"ContainerStarted","Data":"0e3aa74015e80eb1539ee4bc030522a31bc1a2bbd1bdd7711a60eda12e8435b5"}
Sep 16 18:22:09.421144 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:09.421084    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q" podStartSLOduration=3.415648685 podStartE2EDuration="6.421067484s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.005418799s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:22:09.418362957 +0000 UTC m=+3450.781041759" watchObservedRunningTime="2026-09-16 18:22:09.421067484 +0000 UTC m=+3450.783746271"
Sep 16 18:22:10.095008 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:10.094968    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"
Sep 16 18:22:12.189285 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:12.189236    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb" podUID="2216e73f-5855-4c61-80b3-98a47d81f3e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": dial tcp 10.133.0.81:8000: connect: connection refused"
Sep 16 18:22:14.094660 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:14.094616    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"
Sep 16 18:22:14.096556 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:14.096528    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q" podUID="d91c0691-fedf-44a9-9a50-408c6635eca0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 16 18:22:14.166786 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:14.166745    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:22:22.189499 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:22.189453    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb" podUID="2216e73f-5855-4c61-80b3-98a47d81f3e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": dial tcp 10.133.0.81:8000: connect: connection refused"
Sep 16 18:22:24.095102 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:24.095050    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q" podUID="d91c0691-fedf-44a9-9a50-408c6635eca0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 16 18:22:32.188859 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:32.188816    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb" podUID="2216e73f-5855-4c61-80b3-98a47d81f3e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": dial tcp 10.133.0.81:8000: connect: connection refused"
Sep 16 18:22:34.095674 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:34.095625    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q" podUID="d91c0691-fedf-44a9-9a50-408c6635eca0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 16 18:22:34.961816 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:34.961774    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:22:35.018623 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:35.018588    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:22:42.189495 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:42.189448    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb" podUID="2216e73f-5855-4c61-80b3-98a47d81f3e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": dial tcp 10.133.0.81:8000: connect: connection refused"
Sep 16 18:22:42.877934 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:42.877896    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:22:44.095815 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:44.095768    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q" podUID="d91c0691-fedf-44a9-9a50-408c6635eca0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 16 18:22:52.189480 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:52.189432    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb" podUID="2216e73f-5855-4c61-80b3-98a47d81f3e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": dial tcp 10.133.0.81:8000: connect: connection refused"
Sep 16 18:22:54.095385 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:22:54.095331    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q" podUID="d91c0691-fedf-44a9-9a50-408c6635eca0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 16 18:23:02.189638 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:02.189540    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb" podUID="2216e73f-5855-4c61-80b3-98a47d81f3e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": dial tcp 10.133.0.81:8000: connect: connection refused"
Sep 16 18:23:04.094948 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:04.094898    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q" podUID="d91c0691-fedf-44a9-9a50-408c6635eca0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 16 18:23:12.188732 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:12.188681    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb" podUID="2216e73f-5855-4c61-80b3-98a47d81f3e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": dial tcp 10.133.0.81:8000: connect: connection refused"
Sep 16 18:23:14.095062 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:14.095009    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q" podUID="d91c0691-fedf-44a9-9a50-408c6635eca0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 16 18:23:22.188971 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:22.188920    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb" podUID="2216e73f-5855-4c61-80b3-98a47d81f3e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": dial tcp 10.133.0.81:8000: connect: connection refused"
Sep 16 18:23:24.095056 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:24.095012    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q" podUID="d91c0691-fedf-44a9-9a50-408c6635eca0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 16 18:23:32.188675 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:32.188626    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb" podUID="2216e73f-5855-4c61-80b3-98a47d81f3e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": dial tcp 10.133.0.81:8000: connect: connection refused"
Sep 16 18:23:34.105200 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:34.105164    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"
Sep 16 18:23:34.113117 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:34.113090    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"
Sep 16 18:23:35.999322 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:35.999292    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:23:35.999759 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:35.999709    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"]
Sep 16 18:23:36.086319 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:36.086279    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:23:42.198928 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:42.198890    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"
Sep 16 18:23:42.206755 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:42.206725    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"
Sep 16 18:23:46.817538 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:46.817501    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:23:46.833386 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:46.833353    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"]
Sep 16 18:23:46.833774 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:46.833740    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb" podUID="2216e73f-5855-4c61-80b3-98a47d81f3e8" containerName="modelcar" containerID="cri-o://19a7fa1ebd74a095c35ebb56c065f0feb44fe4b1317abc75ea894f4d1873e1a7" gracePeriod=60
Sep 16 18:23:46.915528 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:46.915490    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:23:46.990741 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:46.990713    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb_2216e73f-5855-4c61-80b3-98a47d81f3e8/modelcar/0.log"
Sep 16 18:23:46.991475 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:46.991448    2577 generic.go:356] "Generic (PLEG): container finished" podID="2216e73f-5855-4c61-80b3-98a47d81f3e8" containerID="19a7fa1ebd74a095c35ebb56c065f0feb44fe4b1317abc75ea894f4d1873e1a7" exitCode=143
Sep 16 18:23:46.991628 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:46.991472    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb" event={"ID":"2216e73f-5855-4c61-80b3-98a47d81f3e8","Type":"ContainerDied","Data":"19a7fa1ebd74a095c35ebb56c065f0feb44fe4b1317abc75ea894f4d1873e1a7"}
Sep 16 18:23:51.037963 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:51.037904    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q" podUID="d91c0691-fedf-44a9-9a50-408c6635eca0" containerName="main" containerID="cri-o://0e3aa74015e80eb1539ee4bc030522a31bc1a2bbd1bdd7711a60eda12e8435b5" gracePeriod=45
Sep 16 18:23:51.106666 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:51.106625    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q" podUID="d91c0691-fedf-44a9-9a50-408c6635eca0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 16 18:23:51.686964 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:51.686940    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"
Sep 16 18:23:51.698045 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:51.698020    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d91c0691-fedf-44a9-9a50-408c6635eca0-tmp-dir\") pod \"d91c0691-fedf-44a9-9a50-408c6635eca0\" (UID: \"d91c0691-fedf-44a9-9a50-408c6635eca0\") "
Sep 16 18:23:51.698172 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:51.698054    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d91c0691-fedf-44a9-9a50-408c6635eca0-model-cache\") pod \"d91c0691-fedf-44a9-9a50-408c6635eca0\" (UID: \"d91c0691-fedf-44a9-9a50-408c6635eca0\") "
Sep 16 18:23:51.698172 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:51.698080    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d91c0691-fedf-44a9-9a50-408c6635eca0-cabundle-cert\") pod \"d91c0691-fedf-44a9-9a50-408c6635eca0\" (UID: \"d91c0691-fedf-44a9-9a50-408c6635eca0\") "
Sep 16 18:23:51.698172 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:51.698119    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d91c0691-fedf-44a9-9a50-408c6635eca0-tls-certs\") pod \"d91c0691-fedf-44a9-9a50-408c6635eca0\" (UID: \"d91c0691-fedf-44a9-9a50-408c6635eca0\") "
Sep 16 18:23:51.698172 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:51.698140    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d91c0691-fedf-44a9-9a50-408c6635eca0-kserve-provision-location\") pod \"d91c0691-fedf-44a9-9a50-408c6635eca0\" (UID: \"d91c0691-fedf-44a9-9a50-408c6635eca0\") "
Sep 16 18:23:51.698172 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:51.698169    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d91c0691-fedf-44a9-9a50-408c6635eca0-home\") pod \"d91c0691-fedf-44a9-9a50-408c6635eca0\" (UID: \"d91c0691-fedf-44a9-9a50-408c6635eca0\") "
Sep 16 18:23:51.698440 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:51.698210    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d91c0691-fedf-44a9-9a50-408c6635eca0-dshm\") pod \"d91c0691-fedf-44a9-9a50-408c6635eca0\" (UID: \"d91c0691-fedf-44a9-9a50-408c6635eca0\") "
Sep 16 18:23:51.698440 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:51.698245    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6zqd\" (UniqueName: \"kubernetes.io/projected/d91c0691-fedf-44a9-9a50-408c6635eca0-kube-api-access-p6zqd\") pod \"d91c0691-fedf-44a9-9a50-408c6635eca0\" (UID: \"d91c0691-fedf-44a9-9a50-408c6635eca0\") "
Sep 16 18:23:51.698440 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:51.698320    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91c0691-fedf-44a9-9a50-408c6635eca0-model-cache" (OuterVolumeSpecName: "model-cache") pod "d91c0691-fedf-44a9-9a50-408c6635eca0" (UID: "d91c0691-fedf-44a9-9a50-408c6635eca0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:23:51.698580 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:51.698470    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d91c0691-fedf-44a9-9a50-408c6635eca0-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:23:51.698635 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:51.698615    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91c0691-fedf-44a9-9a50-408c6635eca0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d91c0691-fedf-44a9-9a50-408c6635eca0" (UID: "d91c0691-fedf-44a9-9a50-408c6635eca0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:23:51.698691 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:51.698614    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d91c0691-fedf-44a9-9a50-408c6635eca0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d91c0691-fedf-44a9-9a50-408c6635eca0" (UID: "d91c0691-fedf-44a9-9a50-408c6635eca0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:23:51.698691 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:51.698641    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91c0691-fedf-44a9-9a50-408c6635eca0-home" (OuterVolumeSpecName: "home") pod "d91c0691-fedf-44a9-9a50-408c6635eca0" (UID: "d91c0691-fedf-44a9-9a50-408c6635eca0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:23:51.701113 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:51.701079    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91c0691-fedf-44a9-9a50-408c6635eca0-dshm" (OuterVolumeSpecName: "dshm") pod "d91c0691-fedf-44a9-9a50-408c6635eca0" (UID: "d91c0691-fedf-44a9-9a50-408c6635eca0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:23:51.701311 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:51.701290    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d91c0691-fedf-44a9-9a50-408c6635eca0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d91c0691-fedf-44a9-9a50-408c6635eca0" (UID: "d91c0691-fedf-44a9-9a50-408c6635eca0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:23:51.701379 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:51.701333    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91c0691-fedf-44a9-9a50-408c6635eca0-kube-api-access-p6zqd" (OuterVolumeSpecName: "kube-api-access-p6zqd") pod "d91c0691-fedf-44a9-9a50-408c6635eca0" (UID: "d91c0691-fedf-44a9-9a50-408c6635eca0"). InnerVolumeSpecName "kube-api-access-p6zqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:23:51.715578 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:51.715537    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91c0691-fedf-44a9-9a50-408c6635eca0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d91c0691-fedf-44a9-9a50-408c6635eca0" (UID: "d91c0691-fedf-44a9-9a50-408c6635eca0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:23:51.799273 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:51.799186    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d91c0691-fedf-44a9-9a50-408c6635eca0-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:23:51.799273 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:51.799216    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d91c0691-fedf-44a9-9a50-408c6635eca0-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:23:51.799273 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:51.799226    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6zqd\" (UniqueName: \"kubernetes.io/projected/d91c0691-fedf-44a9-9a50-408c6635eca0-kube-api-access-p6zqd\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:23:51.799273 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:51.799237    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d91c0691-fedf-44a9-9a50-408c6635eca0-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:23:51.799273 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:51.799247    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d91c0691-fedf-44a9-9a50-408c6635eca0-cabundle-cert\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:23:51.799273 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:51.799256    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d91c0691-fedf-44a9-9a50-408c6635eca0-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:23:51.799273 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:51.799265    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d91c0691-fedf-44a9-9a50-408c6635eca0-kserve-provision-location\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:23:52.019902 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:52.019848    2577 generic.go:356] "Generic (PLEG): container finished" podID="d91c0691-fedf-44a9-9a50-408c6635eca0" containerID="0e3aa74015e80eb1539ee4bc030522a31bc1a2bbd1bdd7711a60eda12e8435b5" exitCode=0
Sep 16 18:23:52.020081 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:52.019923    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"
Sep 16 18:23:52.020081 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:52.019930    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q" event={"ID":"d91c0691-fedf-44a9-9a50-408c6635eca0","Type":"ContainerDied","Data":"0e3aa74015e80eb1539ee4bc030522a31bc1a2bbd1bdd7711a60eda12e8435b5"}
Sep 16 18:23:52.020081 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:52.019994    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q" event={"ID":"d91c0691-fedf-44a9-9a50-408c6635eca0","Type":"ContainerDied","Data":"5c23f7a29489a3609f6821fb80f85afb9df0dcbd72c708f5a75b33a5cfe16668"}
Sep 16 18:23:52.020081 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:52.020026    2577 scope.go:117] "RemoveContainer" containerID="0e3aa74015e80eb1539ee4bc030522a31bc1a2bbd1bdd7711a60eda12e8435b5"
Sep 16 18:23:52.033253 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:52.033226    2577 scope.go:117] "RemoveContainer" containerID="45a6ae7e0df370afc69d95e90be7996f04a34446211f3b7207fb32b2bc2a0cc8"
Sep 16 18:23:52.042443 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:52.042288    2577 scope.go:117] "RemoveContainer" containerID="0e3aa74015e80eb1539ee4bc030522a31bc1a2bbd1bdd7711a60eda12e8435b5"
Sep 16 18:23:52.042705 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:23:52.042541    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e3aa74015e80eb1539ee4bc030522a31bc1a2bbd1bdd7711a60eda12e8435b5\": container with ID starting with 0e3aa74015e80eb1539ee4bc030522a31bc1a2bbd1bdd7711a60eda12e8435b5 not found: ID does not exist" containerID="0e3aa74015e80eb1539ee4bc030522a31bc1a2bbd1bdd7711a60eda12e8435b5"
Sep 16 18:23:52.042705 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:52.042568    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e3aa74015e80eb1539ee4bc030522a31bc1a2bbd1bdd7711a60eda12e8435b5"} err="failed to get container status \"0e3aa74015e80eb1539ee4bc030522a31bc1a2bbd1bdd7711a60eda12e8435b5\": rpc error: code = NotFound desc = could not find container \"0e3aa74015e80eb1539ee4bc030522a31bc1a2bbd1bdd7711a60eda12e8435b5\": container with ID starting with 0e3aa74015e80eb1539ee4bc030522a31bc1a2bbd1bdd7711a60eda12e8435b5 not found: ID does not exist"
Sep 16 18:23:52.042705 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:52.042588    2577 scope.go:117] "RemoveContainer" containerID="45a6ae7e0df370afc69d95e90be7996f04a34446211f3b7207fb32b2bc2a0cc8"
Sep 16 18:23:52.042894 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:23:52.042877    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45a6ae7e0df370afc69d95e90be7996f04a34446211f3b7207fb32b2bc2a0cc8\": container with ID starting with 45a6ae7e0df370afc69d95e90be7996f04a34446211f3b7207fb32b2bc2a0cc8 not found: ID does not exist" containerID="45a6ae7e0df370afc69d95e90be7996f04a34446211f3b7207fb32b2bc2a0cc8"
Sep 16 18:23:52.042942 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:52.042898    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a6ae7e0df370afc69d95e90be7996f04a34446211f3b7207fb32b2bc2a0cc8"} err="failed to get container status \"45a6ae7e0df370afc69d95e90be7996f04a34446211f3b7207fb32b2bc2a0cc8\": rpc error: code = NotFound desc = could not find container \"45a6ae7e0df370afc69d95e90be7996f04a34446211f3b7207fb32b2bc2a0cc8\": container with ID starting with 45a6ae7e0df370afc69d95e90be7996f04a34446211f3b7207fb32b2bc2a0cc8 not found: ID does not exist"
Sep 16 18:23:52.052075 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:52.052012    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"]
Sep 16 18:23:52.058808 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:52.058784    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-67897fff-kdp4q"]
Sep 16 18:23:53.080569 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:23:53.080536    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d91c0691-fedf-44a9-9a50-408c6635eca0" path="/var/lib/kubelet/pods/d91c0691-fedf-44a9-9a50-408c6635eca0/volumes"
Sep 16 18:24:01.858154 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:01.858090    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb" podUID="2216e73f-5855-4c61-80b3-98a47d81f3e8" containerName="main" containerID="cri-o://246f6c2daaa132bc07f773753b465799d45668049a290ec33f4d856a5cf00828" gracePeriod=45
Sep 16 18:24:02.200425 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:02.200324    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb" podUID="2216e73f-5855-4c61-80b3-98a47d81f3e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": dial tcp 10.133.0.81:8000: connect: connection refused"
Sep 16 18:24:02.720959 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:02.720937    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb_2216e73f-5855-4c61-80b3-98a47d81f3e8/modelcar/0.log"
Sep 16 18:24:02.723886 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:02.723851    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"
Sep 16 18:24:02.793733 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:02.793640    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2216e73f-5855-4c61-80b3-98a47d81f3e8-home\") pod \"2216e73f-5855-4c61-80b3-98a47d81f3e8\" (UID: \"2216e73f-5855-4c61-80b3-98a47d81f3e8\") "
Sep 16 18:24:02.793733 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:02.793723    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2216e73f-5855-4c61-80b3-98a47d81f3e8-tls-certs\") pod \"2216e73f-5855-4c61-80b3-98a47d81f3e8\" (UID: \"2216e73f-5855-4c61-80b3-98a47d81f3e8\") "
Sep 16 18:24:02.793997 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:02.793750    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7q9jp\" (UniqueName: \"kubernetes.io/projected/2216e73f-5855-4c61-80b3-98a47d81f3e8-kube-api-access-7q9jp\") pod \"2216e73f-5855-4c61-80b3-98a47d81f3e8\" (UID: \"2216e73f-5855-4c61-80b3-98a47d81f3e8\") "
Sep 16 18:24:02.793997 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:02.793786    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2216e73f-5855-4c61-80b3-98a47d81f3e8-dshm\") pod \"2216e73f-5855-4c61-80b3-98a47d81f3e8\" (UID: \"2216e73f-5855-4c61-80b3-98a47d81f3e8\") "
Sep 16 18:24:02.793997 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:02.793905    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2216e73f-5855-4c61-80b3-98a47d81f3e8-kserve-provision-location\") pod \"2216e73f-5855-4c61-80b3-98a47d81f3e8\" (UID: \"2216e73f-5855-4c61-80b3-98a47d81f3e8\") "
Sep 16 18:24:02.793997 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:02.793964    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2216e73f-5855-4c61-80b3-98a47d81f3e8-tmp-dir\") pod \"2216e73f-5855-4c61-80b3-98a47d81f3e8\" (UID: \"2216e73f-5855-4c61-80b3-98a47d81f3e8\") "
Sep 16 18:24:02.793997 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:02.793992    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2216e73f-5855-4c61-80b3-98a47d81f3e8-model-cache\") pod \"2216e73f-5855-4c61-80b3-98a47d81f3e8\" (UID: \"2216e73f-5855-4c61-80b3-98a47d81f3e8\") "
Sep 16 18:24:02.794396 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:02.794345    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2216e73f-5855-4c61-80b3-98a47d81f3e8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2216e73f-5855-4c61-80b3-98a47d81f3e8" (UID: "2216e73f-5855-4c61-80b3-98a47d81f3e8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:24:02.794568 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:02.794495    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2216e73f-5855-4c61-80b3-98a47d81f3e8-model-cache" (OuterVolumeSpecName: "model-cache") pod "2216e73f-5855-4c61-80b3-98a47d81f3e8" (UID: "2216e73f-5855-4c61-80b3-98a47d81f3e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:24:02.794568 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:02.794536    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2216e73f-5855-4c61-80b3-98a47d81f3e8-home" (OuterVolumeSpecName: "home") pod "2216e73f-5855-4c61-80b3-98a47d81f3e8" (UID: "2216e73f-5855-4c61-80b3-98a47d81f3e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:24:02.796122 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:02.796099    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2216e73f-5855-4c61-80b3-98a47d81f3e8-dshm" (OuterVolumeSpecName: "dshm") pod "2216e73f-5855-4c61-80b3-98a47d81f3e8" (UID: "2216e73f-5855-4c61-80b3-98a47d81f3e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:24:02.796225 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:02.796159    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2216e73f-5855-4c61-80b3-98a47d81f3e8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2216e73f-5855-4c61-80b3-98a47d81f3e8" (UID: "2216e73f-5855-4c61-80b3-98a47d81f3e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:24:02.796396 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:02.796374    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2216e73f-5855-4c61-80b3-98a47d81f3e8-kube-api-access-7q9jp" (OuterVolumeSpecName: "kube-api-access-7q9jp") pod "2216e73f-5855-4c61-80b3-98a47d81f3e8" (UID: "2216e73f-5855-4c61-80b3-98a47d81f3e8"). InnerVolumeSpecName "kube-api-access-7q9jp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:24:02.806134 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:02.806108    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2216e73f-5855-4c61-80b3-98a47d81f3e8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2216e73f-5855-4c61-80b3-98a47d81f3e8" (UID: "2216e73f-5855-4c61-80b3-98a47d81f3e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:24:02.895515 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:02.895476    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2216e73f-5855-4c61-80b3-98a47d81f3e8-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:24:02.895515 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:02.895511    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2216e73f-5855-4c61-80b3-98a47d81f3e8-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:24:02.895515 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:02.895522    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2216e73f-5855-4c61-80b3-98a47d81f3e8-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:24:02.895965 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:02.895531    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2216e73f-5855-4c61-80b3-98a47d81f3e8-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:24:02.895965 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:02.895540    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q9jp\" (UniqueName: \"kubernetes.io/projected/2216e73f-5855-4c61-80b3-98a47d81f3e8-kube-api-access-7q9jp\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:24:02.895965 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:02.895549    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2216e73f-5855-4c61-80b3-98a47d81f3e8-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:24:02.895965 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:02.895558    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2216e73f-5855-4c61-80b3-98a47d81f3e8-kserve-provision-location\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:24:03.082796 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:03.082763    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb_2216e73f-5855-4c61-80b3-98a47d81f3e8/modelcar/0.log"
Sep 16 18:24:03.083626 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:03.083601    2577 generic.go:356] "Generic (PLEG): container finished" podID="2216e73f-5855-4c61-80b3-98a47d81f3e8" containerID="246f6c2daaa132bc07f773753b465799d45668049a290ec33f4d856a5cf00828" exitCode=0
Sep 16 18:24:03.083731 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:03.083639    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb" event={"ID":"2216e73f-5855-4c61-80b3-98a47d81f3e8","Type":"ContainerDied","Data":"246f6c2daaa132bc07f773753b465799d45668049a290ec33f4d856a5cf00828"}
Sep 16 18:24:03.083731 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:03.083669    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"
Sep 16 18:24:03.083731 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:03.083696    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb" event={"ID":"2216e73f-5855-4c61-80b3-98a47d81f3e8","Type":"ContainerDied","Data":"ac155eac1631dd590e62dc2f82c1d22ce0a3003e7bd4c266d95ff316aeb91aca"}
Sep 16 18:24:03.083838 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:03.083734    2577 scope.go:117] "RemoveContainer" containerID="19a7fa1ebd74a095c35ebb56c065f0feb44fe4b1317abc75ea894f4d1873e1a7"
Sep 16 18:24:03.095330 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:03.095307    2577 scope.go:117] "RemoveContainer" containerID="246f6c2daaa132bc07f773753b465799d45668049a290ec33f4d856a5cf00828"
Sep 16 18:24:03.103411 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:03.103384    2577 scope.go:117] "RemoveContainer" containerID="bcabfaa52ad87775924f91d8b909d73b2feafe8081420c3d662ed4ce68d97a96"
Sep 16 18:24:03.106978 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:03.106955    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"]
Sep 16 18:24:03.112885 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:03.112853    2577 scope.go:117] "RemoveContainer" containerID="19a7fa1ebd74a095c35ebb56c065f0feb44fe4b1317abc75ea894f4d1873e1a7"
Sep 16 18:24:03.113168 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:24:03.113147    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19a7fa1ebd74a095c35ebb56c065f0feb44fe4b1317abc75ea894f4d1873e1a7\": container with ID starting with 19a7fa1ebd74a095c35ebb56c065f0feb44fe4b1317abc75ea894f4d1873e1a7 not found: ID does not exist" containerID="19a7fa1ebd74a095c35ebb56c065f0feb44fe4b1317abc75ea894f4d1873e1a7"
Sep 16 18:24:03.113168 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:03.113161    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-bdd5fcb98-wsglb"]
Sep 16 18:24:03.113267 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:03.113178    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19a7fa1ebd74a095c35ebb56c065f0feb44fe4b1317abc75ea894f4d1873e1a7"} err="failed to get container status \"19a7fa1ebd74a095c35ebb56c065f0feb44fe4b1317abc75ea894f4d1873e1a7\": rpc error: code = NotFound desc = could not find container \"19a7fa1ebd74a095c35ebb56c065f0feb44fe4b1317abc75ea894f4d1873e1a7\": container with ID starting with 19a7fa1ebd74a095c35ebb56c065f0feb44fe4b1317abc75ea894f4d1873e1a7 not found: ID does not exist"
Sep 16 18:24:03.113267 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:03.113194    2577 scope.go:117] "RemoveContainer" containerID="246f6c2daaa132bc07f773753b465799d45668049a290ec33f4d856a5cf00828"
Sep 16 18:24:03.113436 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:24:03.113417    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"246f6c2daaa132bc07f773753b465799d45668049a290ec33f4d856a5cf00828\": container with ID starting with 246f6c2daaa132bc07f773753b465799d45668049a290ec33f4d856a5cf00828 not found: ID does not exist" containerID="246f6c2daaa132bc07f773753b465799d45668049a290ec33f4d856a5cf00828"
Sep 16 18:24:03.113475 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:03.113441    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"246f6c2daaa132bc07f773753b465799d45668049a290ec33f4d856a5cf00828"} err="failed to get container status \"246f6c2daaa132bc07f773753b465799d45668049a290ec33f4d856a5cf00828\": rpc error: code = NotFound desc = could not find container \"246f6c2daaa132bc07f773753b465799d45668049a290ec33f4d856a5cf00828\": container with ID starting with 246f6c2daaa132bc07f773753b465799d45668049a290ec33f4d856a5cf00828 not found: ID does not exist"
Sep 16 18:24:03.113475 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:03.113457    2577 scope.go:117] "RemoveContainer" containerID="bcabfaa52ad87775924f91d8b909d73b2feafe8081420c3d662ed4ce68d97a96"
Sep 16 18:24:03.113723 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:24:03.113704    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcabfaa52ad87775924f91d8b909d73b2feafe8081420c3d662ed4ce68d97a96\": container with ID starting with bcabfaa52ad87775924f91d8b909d73b2feafe8081420c3d662ed4ce68d97a96 not found: ID does not exist" containerID="bcabfaa52ad87775924f91d8b909d73b2feafe8081420c3d662ed4ce68d97a96"
Sep 16 18:24:03.113787 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:03.113728    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcabfaa52ad87775924f91d8b909d73b2feafe8081420c3d662ed4ce68d97a96"} err="failed to get container status \"bcabfaa52ad87775924f91d8b909d73b2feafe8081420c3d662ed4ce68d97a96\": rpc error: code = NotFound desc = could not find container \"bcabfaa52ad87775924f91d8b909d73b2feafe8081420c3d662ed4ce68d97a96\": container with ID starting with bcabfaa52ad87775924f91d8b909d73b2feafe8081420c3d662ed4ce68d97a96 not found: ID does not exist"
Sep 16 18:24:05.080883 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:05.080832    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2216e73f-5855-4c61-80b3-98a47d81f3e8" path="/var/lib/kubelet/pods/2216e73f-5855-4c61-80b3-98a47d81f3e8/volumes"
Sep 16 18:24:25.585537 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:25.585478    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:27.362497 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:27.362452    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:27.866839 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:27.866796    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:27.980671 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:27.980639    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:28.133092 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:28.132993    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:32.098962 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:32.098852    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:33.832712 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:33.832673    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:33.932682 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:33.932641    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:36.971543 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:36.971491    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:37.102723 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:37.102687    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:38.402786 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.402747    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:38.496822 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.496779    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:38.670614 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.670529    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2"]
Sep 16 18:24:38.670916 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.670901    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2216e73f-5855-4c61-80b3-98a47d81f3e8" containerName="modelcar-init"
Sep 16 18:24:38.670973 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.670917    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2216e73f-5855-4c61-80b3-98a47d81f3e8" containerName="modelcar-init"
Sep 16 18:24:38.670973 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.670929    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2216e73f-5855-4c61-80b3-98a47d81f3e8" containerName="main"
Sep 16 18:24:38.670973 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.670935    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2216e73f-5855-4c61-80b3-98a47d81f3e8" containerName="main"
Sep 16 18:24:38.670973 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.670945    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d91c0691-fedf-44a9-9a50-408c6635eca0" containerName="main"
Sep 16 18:24:38.670973 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.670951    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d91c0691-fedf-44a9-9a50-408c6635eca0" containerName="main"
Sep 16 18:24:38.670973 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.670959    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2216e73f-5855-4c61-80b3-98a47d81f3e8" containerName="modelcar"
Sep 16 18:24:38.670973 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.670964    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2216e73f-5855-4c61-80b3-98a47d81f3e8" containerName="modelcar"
Sep 16 18:24:38.670973 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.670976    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d91c0691-fedf-44a9-9a50-408c6635eca0" containerName="storage-initializer"
Sep 16 18:24:38.671230 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.670982    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d91c0691-fedf-44a9-9a50-408c6635eca0" containerName="storage-initializer"
Sep 16 18:24:38.671230 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.671040    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2216e73f-5855-4c61-80b3-98a47d81f3e8" containerName="modelcar"
Sep 16 18:24:38.671230 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.671048    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2216e73f-5855-4c61-80b3-98a47d81f3e8" containerName="main"
Sep 16 18:24:38.671230 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.671056    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d91c0691-fedf-44a9-9a50-408c6635eca0" containerName="main"
Sep 16 18:24:38.674297 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.674276    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2"
Sep 16 18:24:38.678511 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.678488    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\""
Sep 16 18:24:38.678610 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.678530    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\""
Sep 16 18:24:38.678733 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.678720    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"seaweedfs-s3-creds\""
Sep 16 18:24:38.679294 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.679276    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-rmztm\""
Sep 16 18:24:38.679370 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.679356    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\""
Sep 16 18:24:38.690152 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.690127    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2"]
Sep 16 18:24:38.746419 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.746377    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:38.795454 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.795414    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00173a23-59c3-437a-b148-1a541f992d93-dshm\") pod \"e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2\" (UID: \"00173a23-59c3-437a-b148-1a541f992d93\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2"
Sep 16 18:24:38.795454 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.795452    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00173a23-59c3-437a-b148-1a541f992d93-model-cache\") pod \"e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2\" (UID: \"00173a23-59c3-437a-b148-1a541f992d93\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2"
Sep 16 18:24:38.795687 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.795477    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00173a23-59c3-437a-b148-1a541f992d93-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2\" (UID: \"00173a23-59c3-437a-b148-1a541f992d93\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2"
Sep 16 18:24:38.795687 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.795539    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00173a23-59c3-437a-b148-1a541f992d93-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2\" (UID: \"00173a23-59c3-437a-b148-1a541f992d93\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2"
Sep 16 18:24:38.795687 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.795558    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlxc4\" (UniqueName: \"kubernetes.io/projected/00173a23-59c3-437a-b148-1a541f992d93-kube-api-access-wlxc4\") pod \"e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2\" (UID: \"00173a23-59c3-437a-b148-1a541f992d93\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2"
Sep 16 18:24:38.795687 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.795576    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00173a23-59c3-437a-b148-1a541f992d93-tls-certs\") pod \"e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2\" (UID: \"00173a23-59c3-437a-b148-1a541f992d93\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2"
Sep 16 18:24:38.795845 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.795702    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00173a23-59c3-437a-b148-1a541f992d93-home\") pod \"e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2\" (UID: \"00173a23-59c3-437a-b148-1a541f992d93\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2"
Sep 16 18:24:38.896952 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.896905    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00173a23-59c3-437a-b148-1a541f992d93-home\") pod \"e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2\" (UID: \"00173a23-59c3-437a-b148-1a541f992d93\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2"
Sep 16 18:24:38.897148 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.896984    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00173a23-59c3-437a-b148-1a541f992d93-dshm\") pod \"e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2\" (UID: \"00173a23-59c3-437a-b148-1a541f992d93\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2"
Sep 16 18:24:38.897148 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.897006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00173a23-59c3-437a-b148-1a541f992d93-model-cache\") pod \"e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2\" (UID: \"00173a23-59c3-437a-b148-1a541f992d93\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2"
Sep 16 18:24:38.897148 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.897025    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00173a23-59c3-437a-b148-1a541f992d93-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2\" (UID: \"00173a23-59c3-437a-b148-1a541f992d93\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2"
Sep 16 18:24:38.897148 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.897058    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00173a23-59c3-437a-b148-1a541f992d93-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2\" (UID: \"00173a23-59c3-437a-b148-1a541f992d93\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2"
Sep 16 18:24:38.897148 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.897081    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlxc4\" (UniqueName: \"kubernetes.io/projected/00173a23-59c3-437a-b148-1a541f992d93-kube-api-access-wlxc4\") pod \"e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2\" (UID: \"00173a23-59c3-437a-b148-1a541f992d93\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2"
Sep 16 18:24:38.897148 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.897106    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00173a23-59c3-437a-b148-1a541f992d93-tls-certs\") pod \"e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2\" (UID: \"00173a23-59c3-437a-b148-1a541f992d93\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2"
Sep 16 18:24:38.897477 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.897384    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00173a23-59c3-437a-b148-1a541f992d93-model-cache\") pod \"e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2\" (UID: \"00173a23-59c3-437a-b148-1a541f992d93\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2"
Sep 16 18:24:38.897477 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.897400    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00173a23-59c3-437a-b148-1a541f992d93-home\") pod \"e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2\" (UID: \"00173a23-59c3-437a-b148-1a541f992d93\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2"
Sep 16 18:24:38.897644 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.897620    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00173a23-59c3-437a-b148-1a541f992d93-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2\" (UID: \"00173a23-59c3-437a-b148-1a541f992d93\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2"
Sep 16 18:24:38.897837 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.897815    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00173a23-59c3-437a-b148-1a541f992d93-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2\" (UID: \"00173a23-59c3-437a-b148-1a541f992d93\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2"
Sep 16 18:24:38.899581 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.899547    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00173a23-59c3-437a-b148-1a541f992d93-dshm\") pod \"e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2\" (UID: \"00173a23-59c3-437a-b148-1a541f992d93\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2"
Sep 16 18:24:38.899907 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.899886    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00173a23-59c3-437a-b148-1a541f992d93-tls-certs\") pod \"e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2\" (UID: \"00173a23-59c3-437a-b148-1a541f992d93\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2"
Sep 16 18:24:38.908495 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.908472    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlxc4\" (UniqueName: \"kubernetes.io/projected/00173a23-59c3-437a-b148-1a541f992d93-kube-api-access-wlxc4\") pod \"e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2\" (UID: \"00173a23-59c3-437a-b148-1a541f992d93\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2"
Sep 16 18:24:38.984794 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:38.984702    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2"
Sep 16 18:24:39.148204 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:39.148171    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:39.273122 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:39.273093    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2"]
Sep 16 18:24:39.273928 ip-10-0-136-122 kubenswrapper[2577]: W0916 18:24:39.273901    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00173a23_59c3_437a_b148_1a541f992d93.slice/crio-eded7a4c4c7059b7705316c7beeda5f94e995deb8b5ca08d206bb27984a0e7fb WatchSource:0}: Error finding container eded7a4c4c7059b7705316c7beeda5f94e995deb8b5ca08d206bb27984a0e7fb: Status 404 returned error can't find the container with id eded7a4c4c7059b7705316c7beeda5f94e995deb8b5ca08d206bb27984a0e7fb
Sep 16 18:24:39.279354 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:39.279330    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"seaweedfs-s3-creds\""
Sep 16 18:24:40.279458 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:40.279411    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2" event={"ID":"00173a23-59c3-437a-b148-1a541f992d93","Type":"ContainerStarted","Data":"16fbfe9c956c6084b4931f484620655bca0043c13548d75f889afffcafc3bf37"}
Sep 16 18:24:40.279847 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:40.279516    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2" event={"ID":"00173a23-59c3-437a-b148-1a541f992d93","Type":"ContainerStarted","Data":"eded7a4c4c7059b7705316c7beeda5f94e995deb8b5ca08d206bb27984a0e7fb"}
Sep 16 18:24:43.258721 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:43.258677    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:44.304030 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:44.303995    2577 generic.go:356] "Generic (PLEG): container finished" podID="00173a23-59c3-437a-b148-1a541f992d93" containerID="16fbfe9c956c6084b4931f484620655bca0043c13548d75f889afffcafc3bf37" exitCode=0
Sep 16 18:24:44.304424 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:44.304066    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2" event={"ID":"00173a23-59c3-437a-b148-1a541f992d93","Type":"ContainerDied","Data":"16fbfe9c956c6084b4931f484620655bca0043c13548d75f889afffcafc3bf37"}
Sep 16 18:24:45.311766 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:45.311723    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2" event={"ID":"00173a23-59c3-437a-b148-1a541f992d93","Type":"ContainerStarted","Data":"ddfd2dcd0f25cea3a6e249ac0bf8078c949774be32cc273748e08d438d89c89e"}
Sep 16 18:24:45.338876 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:45.338800    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2" podStartSLOduration=3.020179668 podStartE2EDuration="7.338775142s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.318595474s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:24:45.335132402 +0000 UTC m=+3606.697811188" watchObservedRunningTime="2026-09-16 18:24:45.338775142 +0000 UTC m=+3606.701453927"
Sep 16 18:24:45.985429 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:45.985387    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2"
Sep 16 18:24:48.048957 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.048912    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"]
Sep 16 18:24:48.053564 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.053536    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"
Sep 16 18:24:48.056172 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.056144    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 16 18:24:48.056172 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.056169    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:24:48.056372 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.056226    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 16 18:24:48.056580 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.056564    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Sep 16 18:24:48.056691 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.056605    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 16 18:24:48.056691 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.056636    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-q5hhl\""
Sep 16 18:24:48.066631 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.066606    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"]
Sep 16 18:24:48.078397 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.078370    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebebae06-dad3-413e-911c-801d20799157-cabundle-cert\") pod \"custom-route-timeout-test-kserve-8697bf78bf-mgq8p\" (UID: \"ebebae06-dad3-413e-911c-801d20799157\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"
Sep 16 18:24:48.078716 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.078681    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebebae06-dad3-413e-911c-801d20799157-tls-certs\") pod \"custom-route-timeout-test-kserve-8697bf78bf-mgq8p\" (UID: \"ebebae06-dad3-413e-911c-801d20799157\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"
Sep 16 18:24:48.078837 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.078806    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjf4h\" (UniqueName: \"kubernetes.io/projected/ebebae06-dad3-413e-911c-801d20799157-kube-api-access-pjf4h\") pod \"custom-route-timeout-test-kserve-8697bf78bf-mgq8p\" (UID: \"ebebae06-dad3-413e-911c-801d20799157\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"
Sep 16 18:24:48.078937 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.078851    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebebae06-dad3-413e-911c-801d20799157-home\") pod \"custom-route-timeout-test-kserve-8697bf78bf-mgq8p\" (UID: \"ebebae06-dad3-413e-911c-801d20799157\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"
Sep 16 18:24:48.078937 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.078898    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebebae06-dad3-413e-911c-801d20799157-dshm\") pod \"custom-route-timeout-test-kserve-8697bf78bf-mgq8p\" (UID: \"ebebae06-dad3-413e-911c-801d20799157\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"
Sep 16 18:24:48.079040 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.078953    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebebae06-dad3-413e-911c-801d20799157-tmp-dir\") pod \"custom-route-timeout-test-kserve-8697bf78bf-mgq8p\" (UID: \"ebebae06-dad3-413e-911c-801d20799157\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"
Sep 16 18:24:48.079079 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.079042    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebebae06-dad3-413e-911c-801d20799157-model-cache\") pod \"custom-route-timeout-test-kserve-8697bf78bf-mgq8p\" (UID: \"ebebae06-dad3-413e-911c-801d20799157\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"
Sep 16 18:24:48.079128 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.079075    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebebae06-dad3-413e-911c-801d20799157-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-8697bf78bf-mgq8p\" (UID: \"ebebae06-dad3-413e-911c-801d20799157\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"
Sep 16 18:24:48.083272 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.083244    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:48.180030 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.179996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebebae06-dad3-413e-911c-801d20799157-model-cache\") pod \"custom-route-timeout-test-kserve-8697bf78bf-mgq8p\" (UID: \"ebebae06-dad3-413e-911c-801d20799157\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"
Sep 16 18:24:48.180203 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.180036    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebebae06-dad3-413e-911c-801d20799157-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-8697bf78bf-mgq8p\" (UID: \"ebebae06-dad3-413e-911c-801d20799157\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"
Sep 16 18:24:48.180203 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.180157    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebebae06-dad3-413e-911c-801d20799157-cabundle-cert\") pod \"custom-route-timeout-test-kserve-8697bf78bf-mgq8p\" (UID: \"ebebae06-dad3-413e-911c-801d20799157\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"
Sep 16 18:24:48.180203 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.180188    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebebae06-dad3-413e-911c-801d20799157-tls-certs\") pod \"custom-route-timeout-test-kserve-8697bf78bf-mgq8p\" (UID: \"ebebae06-dad3-413e-911c-801d20799157\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"
Sep 16 18:24:48.180400 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.180229    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjf4h\" (UniqueName: \"kubernetes.io/projected/ebebae06-dad3-413e-911c-801d20799157-kube-api-access-pjf4h\") pod \"custom-route-timeout-test-kserve-8697bf78bf-mgq8p\" (UID: \"ebebae06-dad3-413e-911c-801d20799157\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"
Sep 16 18:24:48.180400 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.180256    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebebae06-dad3-413e-911c-801d20799157-home\") pod \"custom-route-timeout-test-kserve-8697bf78bf-mgq8p\" (UID: \"ebebae06-dad3-413e-911c-801d20799157\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"
Sep 16 18:24:48.180400 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.180284    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebebae06-dad3-413e-911c-801d20799157-dshm\") pod \"custom-route-timeout-test-kserve-8697bf78bf-mgq8p\" (UID: \"ebebae06-dad3-413e-911c-801d20799157\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"
Sep 16 18:24:48.180400 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.180309    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebebae06-dad3-413e-911c-801d20799157-tmp-dir\") pod \"custom-route-timeout-test-kserve-8697bf78bf-mgq8p\" (UID: \"ebebae06-dad3-413e-911c-801d20799157\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"
Sep 16 18:24:48.180615 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.180477    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebebae06-dad3-413e-911c-801d20799157-model-cache\") pod \"custom-route-timeout-test-kserve-8697bf78bf-mgq8p\" (UID: \"ebebae06-dad3-413e-911c-801d20799157\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"
Sep 16 18:24:48.180615 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.180509    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebebae06-dad3-413e-911c-801d20799157-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-8697bf78bf-mgq8p\" (UID: \"ebebae06-dad3-413e-911c-801d20799157\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"
Sep 16 18:24:48.180731 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.180657    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebebae06-dad3-413e-911c-801d20799157-tmp-dir\") pod \"custom-route-timeout-test-kserve-8697bf78bf-mgq8p\" (UID: \"ebebae06-dad3-413e-911c-801d20799157\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"
Sep 16 18:24:48.180905 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.180878    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebebae06-dad3-413e-911c-801d20799157-home\") pod \"custom-route-timeout-test-kserve-8697bf78bf-mgq8p\" (UID: \"ebebae06-dad3-413e-911c-801d20799157\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"
Sep 16 18:24:48.181034 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.181008    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebebae06-dad3-413e-911c-801d20799157-cabundle-cert\") pod \"custom-route-timeout-test-kserve-8697bf78bf-mgq8p\" (UID: \"ebebae06-dad3-413e-911c-801d20799157\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"
Sep 16 18:24:48.182929 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.182905    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebebae06-dad3-413e-911c-801d20799157-dshm\") pod \"custom-route-timeout-test-kserve-8697bf78bf-mgq8p\" (UID: \"ebebae06-dad3-413e-911c-801d20799157\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"
Sep 16 18:24:48.183155 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.183137    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebebae06-dad3-413e-911c-801d20799157-tls-certs\") pod \"custom-route-timeout-test-kserve-8697bf78bf-mgq8p\" (UID: \"ebebae06-dad3-413e-911c-801d20799157\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"
Sep 16 18:24:48.191141 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.191120    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjf4h\" (UniqueName: \"kubernetes.io/projected/ebebae06-dad3-413e-911c-801d20799157-kube-api-access-pjf4h\") pod \"custom-route-timeout-test-kserve-8697bf78bf-mgq8p\" (UID: \"ebebae06-dad3-413e-911c-801d20799157\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"
Sep 16 18:24:48.328290 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.328255    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:48.366170 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.366126    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"
Sep 16 18:24:48.653173 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.653145    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"]
Sep 16 18:24:48.655215 ip-10-0-136-122 kubenswrapper[2577]: W0916 18:24:48.655186    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebebae06_dad3_413e_911c_801d20799157.slice/crio-f8e1dd5edded4b5f1d7f3eee1fa7165330ba6fda2327b1cd7f11ff67a105c5f1 WatchSource:0}: Error finding container f8e1dd5edded4b5f1d7f3eee1fa7165330ba6fda2327b1cd7f11ff67a105c5f1: Status 404 returned error can't find the container with id f8e1dd5edded4b5f1d7f3eee1fa7165330ba6fda2327b1cd7f11ff67a105c5f1
Sep 16 18:24:48.985436 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.985333    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2"
Sep 16 18:24:48.987133 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:48.987102    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2" podUID="00173a23-59c3-437a-b148-1a541f992d93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": dial tcp 10.133.0.83:8000: connect: connection refused"
Sep 16 18:24:49.336952 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:49.336906    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p" event={"ID":"ebebae06-dad3-413e-911c-801d20799157","Type":"ContainerStarted","Data":"d402e757f35cfdbe81a0ea7291fcee0152a08831765c6ffb3f01959b571ca3f7"}
Sep 16 18:24:49.337424 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:49.336977    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p" event={"ID":"ebebae06-dad3-413e-911c-801d20799157","Type":"ContainerStarted","Data":"f8e1dd5edded4b5f1d7f3eee1fa7165330ba6fda2327b1cd7f11ff67a105c5f1"}
Sep 16 18:24:49.944252 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:49.944213    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:50.044038 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:50.043997    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:50.868353 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:50.868318    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:50.868951 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:50.868831    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2"]
Sep 16 18:24:50.968538 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:50.968498    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:51.349790 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:51.349753    2577 generic.go:356] "Generic (PLEG): container finished" podID="ebebae06-dad3-413e-911c-801d20799157" containerID="d402e757f35cfdbe81a0ea7291fcee0152a08831765c6ffb3f01959b571ca3f7" exitCode=0
Sep 16 18:24:51.349790 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:51.349794    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p" event={"ID":"ebebae06-dad3-413e-911c-801d20799157","Type":"ContainerDied","Data":"d402e757f35cfdbe81a0ea7291fcee0152a08831765c6ffb3f01959b571ca3f7"}
Sep 16 18:24:52.359942 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:52.359906    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p" event={"ID":"ebebae06-dad3-413e-911c-801d20799157","Type":"ContainerStarted","Data":"1a161cebaa3a0d60d5f9c04068709b0b92fe18c1ff07d66c446f5d7d13483e87"}
Sep 16 18:24:52.366366 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:52.366326    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"
Sep 16 18:24:52.388632 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:52.388573    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p" podStartSLOduration=1.928709064 podStartE2EDuration="4.388553025s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.459843961s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:24:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:24:52.384748955 +0000 UTC m=+3613.747427742" watchObservedRunningTime="2026-09-16 18:24:52.388553025 +0000 UTC m=+3613.751231811"
Sep 16 18:24:52.465688 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:52.465651    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:52.566795 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:52.566748    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:52.672251 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:52.672145    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:56.815217 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:56.815171    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:56.922042 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:56.922002    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:57.647517 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:57.647478    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:24:58.366903 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:58.366846    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"
Sep 16 18:24:58.368308 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:24:58.368269    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p" podUID="ebebae06-dad3-413e-911c-801d20799157" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": dial tcp 10.133.0.84:8000: connect: connection refused"
Sep 16 18:25:01.669196 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:01.669149    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:01.912188 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:01.912141    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:02.014656 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:02.014551    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:03.608362 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:03.608323    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:03.714393 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:03.714337    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:05.895299 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:05.895252    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2" podUID="00173a23-59c3-437a-b148-1a541f992d93" containerName="main" containerID="cri-o://ddfd2dcd0f25cea3a6e249ac0bf8078c949774be32cc273748e08d438d89c89e" gracePeriod=45
Sep 16 18:25:06.977042 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:06.977007    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:07.068409 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:07.068367    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:08.213062 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:08.213010    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:08.313359 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:08.313323    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:08.367181 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:08.367129    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p" podUID="ebebae06-dad3-413e-911c-801d20799157" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": dial tcp 10.133.0.84:8000: connect: connection refused"
Sep 16 18:25:13.011694 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:13.011648    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:13.078582 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:13.078540    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:15.203571 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:15.203533    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:18.366684 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:18.366630    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p" podUID="ebebae06-dad3-413e-911c-801d20799157" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": dial tcp 10.133.0.84:8000: connect: connection refused"
Sep 16 18:25:28.366934 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:28.366892    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p" podUID="ebebae06-dad3-413e-911c-801d20799157" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": dial tcp 10.133.0.84:8000: connect: connection refused"
Sep 16 18:25:30.100623 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:30.100583    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:30.350049 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:30.350013    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:38.366879 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:38.366821    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p" podUID="ebebae06-dad3-413e-911c-801d20799157" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": dial tcp 10.133.0.84:8000: connect: connection refused"
Sep 16 18:25:43.181931 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:43.181841    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:25:48.367233 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:48.367189    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p" podUID="ebebae06-dad3-413e-911c-801d20799157" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": dial tcp 10.133.0.84:8000: connect: connection refused"
Sep 16 18:25:51.557491 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.557461    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-config-deletion-blocked-when-referenced-0f029b85_e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2_00173a23-59c3-437a-b148-1a541f992d93/main/0.log"
Sep 16 18:25:51.560164 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.560141    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2"
Sep 16 18:25:51.665779 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.665687    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlxc4\" (UniqueName: \"kubernetes.io/projected/00173a23-59c3-437a-b148-1a541f992d93-kube-api-access-wlxc4\") pod \"00173a23-59c3-437a-b148-1a541f992d93\" (UID: \"00173a23-59c3-437a-b148-1a541f992d93\") "
Sep 16 18:25:51.665779 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.665758    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00173a23-59c3-437a-b148-1a541f992d93-tmp-dir\") pod \"00173a23-59c3-437a-b148-1a541f992d93\" (UID: \"00173a23-59c3-437a-b148-1a541f992d93\") "
Sep 16 18:25:51.665779 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.665776    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00173a23-59c3-437a-b148-1a541f992d93-kserve-provision-location\") pod \"00173a23-59c3-437a-b148-1a541f992d93\" (UID: \"00173a23-59c3-437a-b148-1a541f992d93\") "
Sep 16 18:25:51.666094 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.665815    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00173a23-59c3-437a-b148-1a541f992d93-tls-certs\") pod \"00173a23-59c3-437a-b148-1a541f992d93\" (UID: \"00173a23-59c3-437a-b148-1a541f992d93\") "
Sep 16 18:25:51.666094 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.665830    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00173a23-59c3-437a-b148-1a541f992d93-home\") pod \"00173a23-59c3-437a-b148-1a541f992d93\" (UID: \"00173a23-59c3-437a-b148-1a541f992d93\") "
Sep 16 18:25:51.666094 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.665850    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00173a23-59c3-437a-b148-1a541f992d93-dshm\") pod \"00173a23-59c3-437a-b148-1a541f992d93\" (UID: \"00173a23-59c3-437a-b148-1a541f992d93\") "
Sep 16 18:25:51.666094 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.665902    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00173a23-59c3-437a-b148-1a541f992d93-model-cache\") pod \"00173a23-59c3-437a-b148-1a541f992d93\" (UID: \"00173a23-59c3-437a-b148-1a541f992d93\") "
Sep 16 18:25:51.666457 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.666409    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00173a23-59c3-437a-b148-1a541f992d93-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "00173a23-59c3-437a-b148-1a541f992d93" (UID: "00173a23-59c3-437a-b148-1a541f992d93"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:25:51.666457 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.666432    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00173a23-59c3-437a-b148-1a541f992d93-model-cache" (OuterVolumeSpecName: "model-cache") pod "00173a23-59c3-437a-b148-1a541f992d93" (UID: "00173a23-59c3-437a-b148-1a541f992d93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:25:51.666712 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.666447    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00173a23-59c3-437a-b148-1a541f992d93-home" (OuterVolumeSpecName: "home") pod "00173a23-59c3-437a-b148-1a541f992d93" (UID: "00173a23-59c3-437a-b148-1a541f992d93"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:25:51.668446 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.668413    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00173a23-59c3-437a-b148-1a541f992d93-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "00173a23-59c3-437a-b148-1a541f992d93" (UID: "00173a23-59c3-437a-b148-1a541f992d93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:25:51.668446 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.668418    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00173a23-59c3-437a-b148-1a541f992d93-dshm" (OuterVolumeSpecName: "dshm") pod "00173a23-59c3-437a-b148-1a541f992d93" (UID: "00173a23-59c3-437a-b148-1a541f992d93"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:25:51.668632 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.668477    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00173a23-59c3-437a-b148-1a541f992d93-kube-api-access-wlxc4" (OuterVolumeSpecName: "kube-api-access-wlxc4") pod "00173a23-59c3-437a-b148-1a541f992d93" (UID: "00173a23-59c3-437a-b148-1a541f992d93"). InnerVolumeSpecName "kube-api-access-wlxc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:25:51.730511 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.730479    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-config-deletion-blocked-when-referenced-0f029b85_e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2_00173a23-59c3-437a-b148-1a541f992d93/main/0.log"
Sep 16 18:25:51.730971 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.730942    2577 generic.go:356] "Generic (PLEG): container finished" podID="00173a23-59c3-437a-b148-1a541f992d93" containerID="ddfd2dcd0f25cea3a6e249ac0bf8078c949774be32cc273748e08d438d89c89e" exitCode=137
Sep 16 18:25:51.731103 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.731010    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2"
Sep 16 18:25:51.731103 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.731018    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2" event={"ID":"00173a23-59c3-437a-b148-1a541f992d93","Type":"ContainerDied","Data":"ddfd2dcd0f25cea3a6e249ac0bf8078c949774be32cc273748e08d438d89c89e"}
Sep 16 18:25:51.731103 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.731092    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2" event={"ID":"00173a23-59c3-437a-b148-1a541f992d93","Type":"ContainerDied","Data":"eded7a4c4c7059b7705316c7beeda5f94e995deb8b5ca08d206bb27984a0e7fb"}
Sep 16 18:25:51.731282 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.731117    2577 scope.go:117] "RemoveContainer" containerID="ddfd2dcd0f25cea3a6e249ac0bf8078c949774be32cc273748e08d438d89c89e"
Sep 16 18:25:51.732740 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.732716    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00173a23-59c3-437a-b148-1a541f992d93-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "00173a23-59c3-437a-b148-1a541f992d93" (UID: "00173a23-59c3-437a-b148-1a541f992d93"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:25:51.740277 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.740254    2577 scope.go:117] "RemoveContainer" containerID="16fbfe9c956c6084b4931f484620655bca0043c13548d75f889afffcafc3bf37"
Sep 16 18:25:51.748910 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.748889    2577 scope.go:117] "RemoveContainer" containerID="ddfd2dcd0f25cea3a6e249ac0bf8078c949774be32cc273748e08d438d89c89e"
Sep 16 18:25:51.749176 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:25:51.749158    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddfd2dcd0f25cea3a6e249ac0bf8078c949774be32cc273748e08d438d89c89e\": container with ID starting with ddfd2dcd0f25cea3a6e249ac0bf8078c949774be32cc273748e08d438d89c89e not found: ID does not exist" containerID="ddfd2dcd0f25cea3a6e249ac0bf8078c949774be32cc273748e08d438d89c89e"
Sep 16 18:25:51.749264 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.749182    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddfd2dcd0f25cea3a6e249ac0bf8078c949774be32cc273748e08d438d89c89e"} err="failed to get container status \"ddfd2dcd0f25cea3a6e249ac0bf8078c949774be32cc273748e08d438d89c89e\": rpc error: code = NotFound desc = could not find container \"ddfd2dcd0f25cea3a6e249ac0bf8078c949774be32cc273748e08d438d89c89e\": container with ID starting with ddfd2dcd0f25cea3a6e249ac0bf8078c949774be32cc273748e08d438d89c89e not found: ID does not exist"
Sep 16 18:25:51.749264 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.749201    2577 scope.go:117] "RemoveContainer" containerID="16fbfe9c956c6084b4931f484620655bca0043c13548d75f889afffcafc3bf37"
Sep 16 18:25:51.749443 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:25:51.749421    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16fbfe9c956c6084b4931f484620655bca0043c13548d75f889afffcafc3bf37\": container with ID starting with 16fbfe9c956c6084b4931f484620655bca0043c13548d75f889afffcafc3bf37 not found: ID does not exist" containerID="16fbfe9c956c6084b4931f484620655bca0043c13548d75f889afffcafc3bf37"
Sep 16 18:25:51.749505 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.749449    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16fbfe9c956c6084b4931f484620655bca0043c13548d75f889afffcafc3bf37"} err="failed to get container status \"16fbfe9c956c6084b4931f484620655bca0043c13548d75f889afffcafc3bf37\": rpc error: code = NotFound desc = could not find container \"16fbfe9c956c6084b4931f484620655bca0043c13548d75f889afffcafc3bf37\": container with ID starting with 16fbfe9c956c6084b4931f484620655bca0043c13548d75f889afffcafc3bf37 not found: ID does not exist"
Sep 16 18:25:51.767017 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.766985    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00173a23-59c3-437a-b148-1a541f992d93-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:25:51.767017 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.767007    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00173a23-59c3-437a-b148-1a541f992d93-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:25:51.767017 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.767016    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00173a23-59c3-437a-b148-1a541f992d93-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:25:51.767017 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.767023    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00173a23-59c3-437a-b148-1a541f992d93-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:25:51.767273 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.767032    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlxc4\" (UniqueName: \"kubernetes.io/projected/00173a23-59c3-437a-b148-1a541f992d93-kube-api-access-wlxc4\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:25:51.767273 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.767058    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00173a23-59c3-437a-b148-1a541f992d93-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:25:51.767273 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:51.767069    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00173a23-59c3-437a-b148-1a541f992d93-kserve-provision-location\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:25:52.056277 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:52.056245    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2"]
Sep 16 18:25:52.062382 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:52.062353    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6fb6b584d4-nxcq2"]
Sep 16 18:25:53.081475 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:53.081440    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00173a23-59c3-437a-b148-1a541f992d93" path="/var/lib/kubelet/pods/00173a23-59c3-437a-b148-1a541f992d93/volumes"
Sep 16 18:25:58.367436 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:25:58.367389    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p" podUID="ebebae06-dad3-413e-911c-801d20799157" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": dial tcp 10.133.0.84:8000: connect: connection refused"
Sep 16 18:26:04.717496 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:26:04.717455    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:04.816829 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:26:04.816789    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:05.746716 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:26:05.746673    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:08.366777 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:26:08.366737    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p" podUID="ebebae06-dad3-413e-911c-801d20799157" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": dial tcp 10.133.0.84:8000: connect: connection refused"
Sep 16 18:26:18.367072 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:26:18.367021    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p" podUID="ebebae06-dad3-413e-911c-801d20799157" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": dial tcp 10.133.0.84:8000: connect: connection refused"
Sep 16 18:26:28.367017 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:26:28.366970    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p" podUID="ebebae06-dad3-413e-911c-801d20799157" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": dial tcp 10.133.0.84:8000: connect: connection refused"
Sep 16 18:26:38.367091 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:26:38.367042    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p" podUID="ebebae06-dad3-413e-911c-801d20799157" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": dial tcp 10.133.0.84:8000: connect: connection refused"
Sep 16 18:26:48.377322 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:26:48.377284    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"
Sep 16 18:26:48.384878 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:26:48.384843    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"
Sep 16 18:26:53.387593 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:26:53.387550    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:26:53.404757 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:26:53.404727    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"]
Sep 16 18:26:53.483553 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:26:53.483507    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:27:08.427634 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:08.427575    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p" podUID="ebebae06-dad3-413e-911c-801d20799157" containerName="main" containerID="cri-o://1a161cebaa3a0d60d5f9c04068709b0b92fe18c1ff07d66c446f5d7d13483e87" gracePeriod=45
Sep 16 18:27:09.155337 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:09.155303    2577 generic.go:356] "Generic (PLEG): container finished" podID="ebebae06-dad3-413e-911c-801d20799157" containerID="1a161cebaa3a0d60d5f9c04068709b0b92fe18c1ff07d66c446f5d7d13483e87" exitCode=0
Sep 16 18:27:09.155539 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:09.155373    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p" event={"ID":"ebebae06-dad3-413e-911c-801d20799157","Type":"ContainerDied","Data":"1a161cebaa3a0d60d5f9c04068709b0b92fe18c1ff07d66c446f5d7d13483e87"}
Sep 16 18:27:09.184678 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:09.184652    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"
Sep 16 18:27:09.354184 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:09.354132    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebebae06-dad3-413e-911c-801d20799157-home\") pod \"ebebae06-dad3-413e-911c-801d20799157\" (UID: \"ebebae06-dad3-413e-911c-801d20799157\") "
Sep 16 18:27:09.354184 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:09.354197    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebebae06-dad3-413e-911c-801d20799157-tmp-dir\") pod \"ebebae06-dad3-413e-911c-801d20799157\" (UID: \"ebebae06-dad3-413e-911c-801d20799157\") "
Sep 16 18:27:09.354568 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:09.354232    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebebae06-dad3-413e-911c-801d20799157-tls-certs\") pod \"ebebae06-dad3-413e-911c-801d20799157\" (UID: \"ebebae06-dad3-413e-911c-801d20799157\") "
Sep 16 18:27:09.354568 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:09.354250    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjf4h\" (UniqueName: \"kubernetes.io/projected/ebebae06-dad3-413e-911c-801d20799157-kube-api-access-pjf4h\") pod \"ebebae06-dad3-413e-911c-801d20799157\" (UID: \"ebebae06-dad3-413e-911c-801d20799157\") "
Sep 16 18:27:09.354568 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:09.354274    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebebae06-dad3-413e-911c-801d20799157-model-cache\") pod \"ebebae06-dad3-413e-911c-801d20799157\" (UID: \"ebebae06-dad3-413e-911c-801d20799157\") "
Sep 16 18:27:09.354568 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:09.354302    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebebae06-dad3-413e-911c-801d20799157-cabundle-cert\") pod \"ebebae06-dad3-413e-911c-801d20799157\" (UID: \"ebebae06-dad3-413e-911c-801d20799157\") "
Sep 16 18:27:09.354568 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:09.354327    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebebae06-dad3-413e-911c-801d20799157-dshm\") pod \"ebebae06-dad3-413e-911c-801d20799157\" (UID: \"ebebae06-dad3-413e-911c-801d20799157\") "
Sep 16 18:27:09.354568 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:09.354390    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebebae06-dad3-413e-911c-801d20799157-kserve-provision-location\") pod \"ebebae06-dad3-413e-911c-801d20799157\" (UID: \"ebebae06-dad3-413e-911c-801d20799157\") "
Sep 16 18:27:09.354937 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:09.354780    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebebae06-dad3-413e-911c-801d20799157-model-cache" (OuterVolumeSpecName: "model-cache") pod "ebebae06-dad3-413e-911c-801d20799157" (UID: "ebebae06-dad3-413e-911c-801d20799157"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:27:09.355025 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:09.354998    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebebae06-dad3-413e-911c-801d20799157-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ebebae06-dad3-413e-911c-801d20799157" (UID: "ebebae06-dad3-413e-911c-801d20799157"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:27:09.355126 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:09.355093    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebebae06-dad3-413e-911c-801d20799157-home" (OuterVolumeSpecName: "home") pod "ebebae06-dad3-413e-911c-801d20799157" (UID: "ebebae06-dad3-413e-911c-801d20799157"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:27:09.357256 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:09.357230    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebebae06-dad3-413e-911c-801d20799157-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ebebae06-dad3-413e-911c-801d20799157" (UID: "ebebae06-dad3-413e-911c-801d20799157"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:27:09.357256 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:09.357245    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebebae06-dad3-413e-911c-801d20799157-kube-api-access-pjf4h" (OuterVolumeSpecName: "kube-api-access-pjf4h") pod "ebebae06-dad3-413e-911c-801d20799157" (UID: "ebebae06-dad3-413e-911c-801d20799157"). InnerVolumeSpecName "kube-api-access-pjf4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:27:09.357410 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:09.357284    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebebae06-dad3-413e-911c-801d20799157-dshm" (OuterVolumeSpecName: "dshm") pod "ebebae06-dad3-413e-911c-801d20799157" (UID: "ebebae06-dad3-413e-911c-801d20799157"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:27:09.367990 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:09.367957    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebebae06-dad3-413e-911c-801d20799157-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ebebae06-dad3-413e-911c-801d20799157" (UID: "ebebae06-dad3-413e-911c-801d20799157"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:27:09.376436 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:09.376411    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebebae06-dad3-413e-911c-801d20799157-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ebebae06-dad3-413e-911c-801d20799157" (UID: "ebebae06-dad3-413e-911c-801d20799157"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:27:09.455393 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:09.455348    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebebae06-dad3-413e-911c-801d20799157-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:27:09.455393 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:09.455387    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjf4h\" (UniqueName: \"kubernetes.io/projected/ebebae06-dad3-413e-911c-801d20799157-kube-api-access-pjf4h\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:27:09.455393 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:09.455399    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebebae06-dad3-413e-911c-801d20799157-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:27:09.455838 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:09.455410    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebebae06-dad3-413e-911c-801d20799157-cabundle-cert\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:27:09.455838 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:09.455423    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebebae06-dad3-413e-911c-801d20799157-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:27:09.455838 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:09.455431    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebebae06-dad3-413e-911c-801d20799157-kserve-provision-location\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:27:09.455838 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:09.455439    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebebae06-dad3-413e-911c-801d20799157-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:27:09.455838 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:09.455448    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebebae06-dad3-413e-911c-801d20799157-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:27:10.162277 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:10.162231    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"
Sep 16 18:27:10.162277 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:10.162243    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p" event={"ID":"ebebae06-dad3-413e-911c-801d20799157","Type":"ContainerDied","Data":"f8e1dd5edded4b5f1d7f3eee1fa7165330ba6fda2327b1cd7f11ff67a105c5f1"}
Sep 16 18:27:10.162516 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:10.162313    2577 scope.go:117] "RemoveContainer" containerID="1a161cebaa3a0d60d5f9c04068709b0b92fe18c1ff07d66c446f5d7d13483e87"
Sep 16 18:27:10.171740 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:10.171718    2577 scope.go:117] "RemoveContainer" containerID="d402e757f35cfdbe81a0ea7291fcee0152a08831765c6ffb3f01959b571ca3f7"
Sep 16 18:27:10.187486 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:10.187462    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"]
Sep 16 18:27:10.193376 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:10.193352    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-8697bf78bf-mgq8p"]
Sep 16 18:27:11.081487 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:11.081447    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebebae06-dad3-413e-911c-801d20799157" path="/var/lib/kubelet/pods/ebebae06-dad3-413e-911c-801d20799157/volumes"
Sep 16 18:27:32.340925 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:32.340804    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:27:32.457428 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:32.457384    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:27:43.514078 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.514034    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:27:43.619186 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.619150    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:27:43.749912 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.749879    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"]
Sep 16 18:27:43.750303 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.750284    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebebae06-dad3-413e-911c-801d20799157" containerName="storage-initializer"
Sep 16 18:27:43.750391 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.750306    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebebae06-dad3-413e-911c-801d20799157" containerName="storage-initializer"
Sep 16 18:27:43.750391 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.750318    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebebae06-dad3-413e-911c-801d20799157" containerName="main"
Sep 16 18:27:43.750391 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.750326    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebebae06-dad3-413e-911c-801d20799157" containerName="main"
Sep 16 18:27:43.750391 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.750340    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00173a23-59c3-437a-b148-1a541f992d93" containerName="main"
Sep 16 18:27:43.750391 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.750348    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="00173a23-59c3-437a-b148-1a541f992d93" containerName="main"
Sep 16 18:27:43.750391 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.750361    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00173a23-59c3-437a-b148-1a541f992d93" containerName="storage-initializer"
Sep 16 18:27:43.750391 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.750369    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="00173a23-59c3-437a-b148-1a541f992d93" containerName="storage-initializer"
Sep 16 18:27:43.750740 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.750459    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="00173a23-59c3-437a-b148-1a541f992d93" containerName="main"
Sep 16 18:27:43.750740 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.750476    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebebae06-dad3-413e-911c-801d20799157" containerName="main"
Sep 16 18:27:43.754847 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.754826    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:43.758378 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.758352    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-55rwh\""
Sep 16 18:27:43.758517 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.758397    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 16 18:27:43.758517 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.758443    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 16 18:27:43.759100 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.759081    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Sep 16 18:27:43.769422 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.769366    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"]
Sep 16 18:27:43.788632 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.788606    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:27:43.823126 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.823089    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-credential-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p29n6\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:43.823272 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.823129    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-istio-token\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p29n6\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:43.823272 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.823169    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-istio-envoy\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p29n6\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:43.823272 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.823224    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-workload-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p29n6\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:43.823459 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.823270    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-workload-certs\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p29n6\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:43.823459 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.823296    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-istio-podinfo\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p29n6\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:43.823459 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.823325    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-istio-data\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p29n6\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:43.823459 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.823364    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p29n6\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:43.823459 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.823422    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62p8b\" (UniqueName: \"kubernetes.io/projected/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-kube-api-access-62p8b\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p29n6\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:43.924906 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.924837    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-credential-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p29n6\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:43.924906 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.924906    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-istio-token\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p29n6\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:43.925167 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.924936    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-istio-envoy\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p29n6\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:43.925167 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.924961    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-workload-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p29n6\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:43.925167 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.924981    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-workload-certs\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p29n6\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:43.925167 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.925001    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-istio-podinfo\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p29n6\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:43.925167 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.925028    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-istio-data\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p29n6\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:43.925167 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.925068    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p29n6\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:43.925167 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.925106    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62p8b\" (UniqueName: \"kubernetes.io/projected/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-kube-api-access-62p8b\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p29n6\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:43.925511 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.925351    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-credential-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p29n6\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:43.925511 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.925422    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-workload-certs\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p29n6\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:43.925585 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.925537    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-istio-data\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p29n6\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:43.925765 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.925742    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-workload-socket\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p29n6\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:43.925962 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.925946    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p29n6\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:43.927207 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.927190    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-istio-envoy\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p29n6\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:43.927584 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.927563    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-istio-podinfo\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p29n6\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:43.929733 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.929688    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:27:43.936989 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.936967    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-istio-token\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p29n6\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:43.937229 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:43.937211    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62p8b\" (UniqueName: \"kubernetes.io/projected/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-kube-api-access-62p8b\") pod \"router-gateway-1-openshift-default-56bd58cf4d-p29n6\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:44.065795 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:44.065762    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:44.347130 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:44.347102    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"]
Sep 16 18:27:44.348988 ip-10-0-136-122 kubenswrapper[2577]: W0916 18:27:44.348962    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf3fcfd_2a37_4fef_b2f1_1299f26ddc86.slice/crio-f820fe4ea4168c3e6cb4d17d888a73c8776c9f16eedad32ff5b978ecfd224fa8 WatchSource:0}: Error finding container f820fe4ea4168c3e6cb4d17d888a73c8776c9f16eedad32ff5b978ecfd224fa8: Status 404 returned error can't find the container with id f820fe4ea4168c3e6cb4d17d888a73c8776c9f16eedad32ff5b978ecfd224fa8
Sep 16 18:27:44.351152 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:44.351117    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:27:44.351239 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:44.351191    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:27:44.351239 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:44.351224    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:27:45.348474 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:45.348434    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6" event={"ID":"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86","Type":"ContainerStarted","Data":"51f62b4bfc5027301af942694661f709fc4752980bb0596e4a4183c84e4d0583"}
Sep 16 18:27:45.348924 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:45.348495    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6" event={"ID":"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86","Type":"ContainerStarted","Data":"f820fe4ea4168c3e6cb4d17d888a73c8776c9f16eedad32ff5b978ecfd224fa8"}
Sep 16 18:27:45.372028 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:45.371977    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6" podStartSLOduration=2.371960035 podStartE2EDuration="2.371960035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:27:45.369643547 +0000 UTC m=+3786.732322322" watchObservedRunningTime="2026-09-16 18:27:45.371960035 +0000 UTC m=+3786.734638812"
Sep 16 18:27:46.066684 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:46.066647    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:46.072253 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:46.072218    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:46.354174 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:46.354074    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:46.355207 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:46.355178    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:47.341231 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:47.341193    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:27:47.626183 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:47.626137    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:27:50.339334 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:50.339298    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:27:56.407059 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:56.407024    2577 generic.go:356] "Generic (PLEG): container finished" podID="daf3fcfd-2a37-4fef-b2f1-1299f26ddc86" containerID="51f62b4bfc5027301af942694661f709fc4752980bb0596e4a4183c84e4d0583" exitCode=0
Sep 16 18:27:56.407578 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:56.407103    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6" event={"ID":"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86","Type":"ContainerDied","Data":"51f62b4bfc5027301af942694661f709fc4752980bb0596e4a4183c84e4d0583"}
Sep 16 18:27:56.407649 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:56.407596    2577 scope.go:117] "RemoveContainer" containerID="51f62b4bfc5027301af942694661f709fc4752980bb0596e4a4183c84e4d0583"
Sep 16 18:27:56.408581 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:56.408549    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:27:56.408646 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:56.408620    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:27:56.408686 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:56.408660    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:27:56.445705 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:56.445662    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:27:56.577538 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:56.577501    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:27:57.418826 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:57.418784    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6" event={"ID":"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86","Type":"ContainerStarted","Data":"f0995ebb10dd0639e68bd6625da78790b41abdf9504ce48ee8231ad9be325fd6"}
Sep 16 18:27:58.066576 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:58.066534    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:27:58.067577 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:58.067545    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6" podUID="daf3fcfd-2a37-4fef-b2f1-1299f26ddc86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Sep 16 18:27:59.066907 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:27:59.066844    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6" podUID="daf3fcfd-2a37-4fef-b2f1-1299f26ddc86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Sep 16 18:28:00.066312 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:00.066263    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6" podUID="daf3fcfd-2a37-4fef-b2f1-1299f26ddc86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Sep 16 18:28:01.066709 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.066667    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6" podUID="daf3fcfd-2a37-4fef-b2f1-1299f26ddc86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Sep 16 18:28:01.096943 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.096902    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"]
Sep 16 18:28:01.124210 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.124176    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"]
Sep 16 18:28:01.124534 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.124517    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"
Sep 16 18:28:01.126261 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.126228    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:01.128857 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.128831    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 16 18:28:01.129178 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.129017    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:28:01.129178 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.129129    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-4s8vf\""
Sep 16 18:28:01.129316 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.129266    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Sep 16 18:28:01.177175 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.177146    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/717990dd-b98d-4baa-92da-ea34409707b4-dshm\") pod \"router-with-refs-test-kserve-7cbb94df-jm269\" (UID: \"717990dd-b98d-4baa-92da-ea34409707b4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"
Sep 16 18:28:01.177301 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.177176    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/717990dd-b98d-4baa-92da-ea34409707b4-model-cache\") pod \"router-with-refs-test-kserve-7cbb94df-jm269\" (UID: \"717990dd-b98d-4baa-92da-ea34409707b4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"
Sep 16 18:28:01.177301 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.177195    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/717990dd-b98d-4baa-92da-ea34409707b4-cabundle-cert\") pod \"router-with-refs-test-kserve-7cbb94df-jm269\" (UID: \"717990dd-b98d-4baa-92da-ea34409707b4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"
Sep 16 18:28:01.177301 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.177213    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/717990dd-b98d-4baa-92da-ea34409707b4-tmp-dir\") pod \"router-with-refs-test-kserve-7cbb94df-jm269\" (UID: \"717990dd-b98d-4baa-92da-ea34409707b4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"
Sep 16 18:28:01.177301 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.177281    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/717990dd-b98d-4baa-92da-ea34409707b4-home\") pod \"router-with-refs-test-kserve-7cbb94df-jm269\" (UID: \"717990dd-b98d-4baa-92da-ea34409707b4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"
Sep 16 18:28:01.177301 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.177298    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/717990dd-b98d-4baa-92da-ea34409707b4-kserve-provision-location\") pod \"router-with-refs-test-kserve-7cbb94df-jm269\" (UID: \"717990dd-b98d-4baa-92da-ea34409707b4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"
Sep 16 18:28:01.177476 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.177316    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/717990dd-b98d-4baa-92da-ea34409707b4-tls-certs\") pod \"router-with-refs-test-kserve-7cbb94df-jm269\" (UID: \"717990dd-b98d-4baa-92da-ea34409707b4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"
Sep 16 18:28:01.177476 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.177340    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7x4r\" (UniqueName: \"kubernetes.io/projected/717990dd-b98d-4baa-92da-ea34409707b4-kube-api-access-q7x4r\") pod \"router-with-refs-test-kserve-7cbb94df-jm269\" (UID: \"717990dd-b98d-4baa-92da-ea34409707b4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"
Sep 16 18:28:01.278458 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.278413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/717990dd-b98d-4baa-92da-ea34409707b4-dshm\") pod \"router-with-refs-test-kserve-7cbb94df-jm269\" (UID: \"717990dd-b98d-4baa-92da-ea34409707b4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"
Sep 16 18:28:01.278458 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.278460    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/717990dd-b98d-4baa-92da-ea34409707b4-model-cache\") pod \"router-with-refs-test-kserve-7cbb94df-jm269\" (UID: \"717990dd-b98d-4baa-92da-ea34409707b4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"
Sep 16 18:28:01.278728 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.278483    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/717990dd-b98d-4baa-92da-ea34409707b4-cabundle-cert\") pod \"router-with-refs-test-kserve-7cbb94df-jm269\" (UID: \"717990dd-b98d-4baa-92da-ea34409707b4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"
Sep 16 18:28:01.278728 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.278505    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/717990dd-b98d-4baa-92da-ea34409707b4-tmp-dir\") pod \"router-with-refs-test-kserve-7cbb94df-jm269\" (UID: \"717990dd-b98d-4baa-92da-ea34409707b4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"
Sep 16 18:28:01.278728 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.278560    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/717990dd-b98d-4baa-92da-ea34409707b4-home\") pod \"router-with-refs-test-kserve-7cbb94df-jm269\" (UID: \"717990dd-b98d-4baa-92da-ea34409707b4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"
Sep 16 18:28:01.278728 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.278582    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/717990dd-b98d-4baa-92da-ea34409707b4-kserve-provision-location\") pod \"router-with-refs-test-kserve-7cbb94df-jm269\" (UID: \"717990dd-b98d-4baa-92da-ea34409707b4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"
Sep 16 18:28:01.278728 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.278614    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/717990dd-b98d-4baa-92da-ea34409707b4-tls-certs\") pod \"router-with-refs-test-kserve-7cbb94df-jm269\" (UID: \"717990dd-b98d-4baa-92da-ea34409707b4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"
Sep 16 18:28:01.278728 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.278652    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7x4r\" (UniqueName: \"kubernetes.io/projected/717990dd-b98d-4baa-92da-ea34409707b4-kube-api-access-q7x4r\") pod \"router-with-refs-test-kserve-7cbb94df-jm269\" (UID: \"717990dd-b98d-4baa-92da-ea34409707b4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"
Sep 16 18:28:01.279072 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.278931    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/717990dd-b98d-4baa-92da-ea34409707b4-model-cache\") pod \"router-with-refs-test-kserve-7cbb94df-jm269\" (UID: \"717990dd-b98d-4baa-92da-ea34409707b4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"
Sep 16 18:28:01.279117 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.279106    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/717990dd-b98d-4baa-92da-ea34409707b4-tmp-dir\") pod \"router-with-refs-test-kserve-7cbb94df-jm269\" (UID: \"717990dd-b98d-4baa-92da-ea34409707b4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"
Sep 16 18:28:01.279166 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.279149    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/717990dd-b98d-4baa-92da-ea34409707b4-home\") pod \"router-with-refs-test-kserve-7cbb94df-jm269\" (UID: \"717990dd-b98d-4baa-92da-ea34409707b4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"
Sep 16 18:28:01.279281 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.279255    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/717990dd-b98d-4baa-92da-ea34409707b4-kserve-provision-location\") pod \"router-with-refs-test-kserve-7cbb94df-jm269\" (UID: \"717990dd-b98d-4baa-92da-ea34409707b4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"
Sep 16 18:28:01.279408 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.279385    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/717990dd-b98d-4baa-92da-ea34409707b4-cabundle-cert\") pod \"router-with-refs-test-kserve-7cbb94df-jm269\" (UID: \"717990dd-b98d-4baa-92da-ea34409707b4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"
Sep 16 18:28:01.280874 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.280831    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/717990dd-b98d-4baa-92da-ea34409707b4-dshm\") pod \"router-with-refs-test-kserve-7cbb94df-jm269\" (UID: \"717990dd-b98d-4baa-92da-ea34409707b4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"
Sep 16 18:28:01.281369 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.281345    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/717990dd-b98d-4baa-92da-ea34409707b4-tls-certs\") pod \"router-with-refs-test-kserve-7cbb94df-jm269\" (UID: \"717990dd-b98d-4baa-92da-ea34409707b4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"
Sep 16 18:28:01.297587 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.297511    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7x4r\" (UniqueName: \"kubernetes.io/projected/717990dd-b98d-4baa-92da-ea34409707b4-kube-api-access-q7x4r\") pod \"router-with-refs-test-kserve-7cbb94df-jm269\" (UID: \"717990dd-b98d-4baa-92da-ea34409707b4\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"
Sep 16 18:28:01.354508 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.354418    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:28:01.372162 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.372128    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:01.436545 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.436505    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"
Sep 16 18:28:01.565343 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.565302    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:01.732689 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:01.732659    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"]
Sep 16 18:28:01.733538 ip-10-0-136-122 kubenswrapper[2577]: W0916 18:28:01.733504    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod717990dd_b98d_4baa_92da_ea34409707b4.slice/crio-ac43f66d16f5999abf1aa4dc2cb26e9e5a9586904e5e05e722bd81570908b96c WatchSource:0}: Error finding container ac43f66d16f5999abf1aa4dc2cb26e9e5a9586904e5e05e722bd81570908b96c: Status 404 returned error can't find the container with id ac43f66d16f5999abf1aa4dc2cb26e9e5a9586904e5e05e722bd81570908b96c
Sep 16 18:28:02.066433 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:02.066387    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6" podUID="daf3fcfd-2a37-4fef-b2f1-1299f26ddc86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Sep 16 18:28:02.447334 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:02.447226    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269" event={"ID":"717990dd-b98d-4baa-92da-ea34409707b4","Type":"ContainerStarted","Data":"23d35b83f816e3e57f1b0e35743cd8cd4df4276813bd73f1bd70a50b29aa43d5"}
Sep 16 18:28:02.447334 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:02.447292    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269" event={"ID":"717990dd-b98d-4baa-92da-ea34409707b4","Type":"ContainerStarted","Data":"ac43f66d16f5999abf1aa4dc2cb26e9e5a9586904e5e05e722bd81570908b96c"}
Sep 16 18:28:03.070549 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:03.070507    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:28:03.071649 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:03.071619    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:28:04.459356 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:04.459271    2577 generic.go:356] "Generic (PLEG): container finished" podID="717990dd-b98d-4baa-92da-ea34409707b4" containerID="23d35b83f816e3e57f1b0e35743cd8cd4df4276813bd73f1bd70a50b29aa43d5" exitCode=0
Sep 16 18:28:04.459356 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:04.459308    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269" event={"ID":"717990dd-b98d-4baa-92da-ea34409707b4","Type":"ContainerDied","Data":"23d35b83f816e3e57f1b0e35743cd8cd4df4276813bd73f1bd70a50b29aa43d5"}
Sep 16 18:28:05.471272 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:05.471224    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269" event={"ID":"717990dd-b98d-4baa-92da-ea34409707b4","Type":"ContainerStarted","Data":"4a7f38ef9bebb29c469a616c4410794f8817cf8ade9082f0c238546d60000921"}
Sep 16 18:28:05.499504 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:05.499450    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269" podStartSLOduration=2.13586117 podStartE2EDuration="4.49943331s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.36357214s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:28:05.496942676 +0000 UTC m=+3806.859621461" watchObservedRunningTime="2026-09-16 18:28:05.49943331 +0000 UTC m=+3806.862112096"
Sep 16 18:28:06.436750 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:06.436709    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"
Sep 16 18:28:11.437462 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:11.437419    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"
Sep 16 18:28:11.439449 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:11.439418    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269" podUID="717990dd-b98d-4baa-92da-ea34409707b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused"
Sep 16 18:28:19.337838 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:19.337791    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:20.762523 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.762484    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"]
Sep 16 18:28:20.767321 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.767298    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"
Sep 16 18:28:20.770066 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.770040    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 16 18:28:20.770229 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.770200    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 16 18:28:20.770313 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.770233    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 16 18:28:20.770380 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.770312    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:28:20.770436 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.770408    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 16 18:28:20.770552 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.770532    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-l868g\""
Sep 16 18:28:20.779150 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.779124    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:20.779498 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.779476    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"]
Sep 16 18:28:20.856915 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.856878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d90b9763-cd69-4879-9274-5f7f5ed9fcac-dshm\") pod \"stop-feature-test-kserve-57d77dbc48-s9hcb\" (UID: \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"
Sep 16 18:28:20.857093 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.856926    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d90b9763-cd69-4879-9274-5f7f5ed9fcac-tls-certs\") pod \"stop-feature-test-kserve-57d77dbc48-s9hcb\" (UID: \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"
Sep 16 18:28:20.857093 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.856965    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d90b9763-cd69-4879-9274-5f7f5ed9fcac-home\") pod \"stop-feature-test-kserve-57d77dbc48-s9hcb\" (UID: \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"
Sep 16 18:28:20.857093 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.856987    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d90b9763-cd69-4879-9274-5f7f5ed9fcac-kserve-provision-location\") pod \"stop-feature-test-kserve-57d77dbc48-s9hcb\" (UID: \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"
Sep 16 18:28:20.857227 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.857103    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnjpk\" (UniqueName: \"kubernetes.io/projected/d90b9763-cd69-4879-9274-5f7f5ed9fcac-kube-api-access-vnjpk\") pod \"stop-feature-test-kserve-57d77dbc48-s9hcb\" (UID: \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"
Sep 16 18:28:20.857227 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.857133    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d90b9763-cd69-4879-9274-5f7f5ed9fcac-cabundle-cert\") pod \"stop-feature-test-kserve-57d77dbc48-s9hcb\" (UID: \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"
Sep 16 18:28:20.857227 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.857158    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d90b9763-cd69-4879-9274-5f7f5ed9fcac-tmp-dir\") pod \"stop-feature-test-kserve-57d77dbc48-s9hcb\" (UID: \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"
Sep 16 18:28:20.857227 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.857207    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d90b9763-cd69-4879-9274-5f7f5ed9fcac-model-cache\") pod \"stop-feature-test-kserve-57d77dbc48-s9hcb\" (UID: \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"
Sep 16 18:28:20.958091 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.958047    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d90b9763-cd69-4879-9274-5f7f5ed9fcac-home\") pod \"stop-feature-test-kserve-57d77dbc48-s9hcb\" (UID: \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"
Sep 16 18:28:20.958291 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.958108    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d90b9763-cd69-4879-9274-5f7f5ed9fcac-kserve-provision-location\") pod \"stop-feature-test-kserve-57d77dbc48-s9hcb\" (UID: \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"
Sep 16 18:28:20.958291 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.958215    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnjpk\" (UniqueName: \"kubernetes.io/projected/d90b9763-cd69-4879-9274-5f7f5ed9fcac-kube-api-access-vnjpk\") pod \"stop-feature-test-kserve-57d77dbc48-s9hcb\" (UID: \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"
Sep 16 18:28:20.958291 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.958247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d90b9763-cd69-4879-9274-5f7f5ed9fcac-cabundle-cert\") pod \"stop-feature-test-kserve-57d77dbc48-s9hcb\" (UID: \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"
Sep 16 18:28:20.958480 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.958389    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d90b9763-cd69-4879-9274-5f7f5ed9fcac-tmp-dir\") pod \"stop-feature-test-kserve-57d77dbc48-s9hcb\" (UID: \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"
Sep 16 18:28:20.958480 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.958441    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d90b9763-cd69-4879-9274-5f7f5ed9fcac-model-cache\") pod \"stop-feature-test-kserve-57d77dbc48-s9hcb\" (UID: \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"
Sep 16 18:28:20.958589 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.958489    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d90b9763-cd69-4879-9274-5f7f5ed9fcac-dshm\") pod \"stop-feature-test-kserve-57d77dbc48-s9hcb\" (UID: \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"
Sep 16 18:28:20.958589 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.958491    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d90b9763-cd69-4879-9274-5f7f5ed9fcac-home\") pod \"stop-feature-test-kserve-57d77dbc48-s9hcb\" (UID: \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"
Sep 16 18:28:20.958589 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.958515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d90b9763-cd69-4879-9274-5f7f5ed9fcac-tls-certs\") pod \"stop-feature-test-kserve-57d77dbc48-s9hcb\" (UID: \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"
Sep 16 18:28:20.958589 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.958543    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d90b9763-cd69-4879-9274-5f7f5ed9fcac-kserve-provision-location\") pod \"stop-feature-test-kserve-57d77dbc48-s9hcb\" (UID: \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"
Sep 16 18:28:20.958781 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.958760    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d90b9763-cd69-4879-9274-5f7f5ed9fcac-tmp-dir\") pod \"stop-feature-test-kserve-57d77dbc48-s9hcb\" (UID: \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"
Sep 16 18:28:20.958835 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.958815    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d90b9763-cd69-4879-9274-5f7f5ed9fcac-model-cache\") pod \"stop-feature-test-kserve-57d77dbc48-s9hcb\" (UID: \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"
Sep 16 18:28:20.959080 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.959051    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d90b9763-cd69-4879-9274-5f7f5ed9fcac-cabundle-cert\") pod \"stop-feature-test-kserve-57d77dbc48-s9hcb\" (UID: \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"
Sep 16 18:28:20.960717 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.960699    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d90b9763-cd69-4879-9274-5f7f5ed9fcac-dshm\") pod \"stop-feature-test-kserve-57d77dbc48-s9hcb\" (UID: \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"
Sep 16 18:28:20.961091 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.961073    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d90b9763-cd69-4879-9274-5f7f5ed9fcac-tls-certs\") pod \"stop-feature-test-kserve-57d77dbc48-s9hcb\" (UID: \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"
Sep 16 18:28:20.967271 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:20.967246    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnjpk\" (UniqueName: \"kubernetes.io/projected/d90b9763-cd69-4879-9274-5f7f5ed9fcac-kube-api-access-vnjpk\") pod \"stop-feature-test-kserve-57d77dbc48-s9hcb\" (UID: \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"
Sep 16 18:28:21.080127 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:21.080091    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"
Sep 16 18:28:21.212102 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:21.212068    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:21.369682 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:21.369649    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"]
Sep 16 18:28:21.369829 ip-10-0-136-122 kubenswrapper[2577]: W0916 18:28:21.369673    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd90b9763_cd69_4879_9274_5f7f5ed9fcac.slice/crio-58ec7b3d3d974f7e8b9a7433905b9264d279b26465c097fd18af492cb51712dc WatchSource:0}: Error finding container 58ec7b3d3d974f7e8b9a7433905b9264d279b26465c097fd18af492cb51712dc: Status 404 returned error can't find the container with id 58ec7b3d3d974f7e8b9a7433905b9264d279b26465c097fd18af492cb51712dc
Sep 16 18:28:21.437697 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:21.437661    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269" podUID="717990dd-b98d-4baa-92da-ea34409707b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused"
Sep 16 18:28:21.566382 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:21.566343    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb" event={"ID":"d90b9763-cd69-4879-9274-5f7f5ed9fcac","Type":"ContainerStarted","Data":"a4f842b93706ca9c04de4ce7d69f129fda68b9346a7e6027f935f818ca5f7c6a"}
Sep 16 18:28:21.566576 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:21.566402    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb" event={"ID":"d90b9763-cd69-4879-9274-5f7f5ed9fcac","Type":"ContainerStarted","Data":"58ec7b3d3d974f7e8b9a7433905b9264d279b26465c097fd18af492cb51712dc"}
Sep 16 18:28:24.585229 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:24.585191    2577 generic.go:356] "Generic (PLEG): container finished" podID="d90b9763-cd69-4879-9274-5f7f5ed9fcac" containerID="a4f842b93706ca9c04de4ce7d69f129fda68b9346a7e6027f935f818ca5f7c6a" exitCode=0
Sep 16 18:28:24.585623 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:24.585247    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb" event={"ID":"d90b9763-cd69-4879-9274-5f7f5ed9fcac","Type":"ContainerDied","Data":"a4f842b93706ca9c04de4ce7d69f129fda68b9346a7e6027f935f818ca5f7c6a"}
Sep 16 18:28:25.597109 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:25.597063    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb" event={"ID":"d90b9763-cd69-4879-9274-5f7f5ed9fcac","Type":"ContainerStarted","Data":"a7c7a29b11c5c731ffbba77f4af3baa47a962443ff22f96712a83b1dd6f570ff"}
Sep 16 18:28:25.624825 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:25.624753    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb" podStartSLOduration=3.209284067 podStartE2EDuration="5.624726382s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.415442315s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:28:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:28:25.621260198 +0000 UTC m=+3826.983938997" watchObservedRunningTime="2026-09-16 18:28:25.624726382 +0000 UTC m=+3826.987405169"
Sep 16 18:28:26.080294 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:26.080255    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"
Sep 16 18:28:31.082344 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:31.082305    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"
Sep 16 18:28:31.082818 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:31.082422    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb" podUID="d90b9763-cd69-4879-9274-5f7f5ed9fcac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 16 18:28:31.437370 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:31.437265    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269" podUID="717990dd-b98d-4baa-92da-ea34409707b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused"
Sep 16 18:28:36.253680 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:36.253642    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:28:41.081056 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:41.081009    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb" podUID="d90b9763-cd69-4879-9274-5f7f5ed9fcac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 16 18:28:41.437596 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:41.437487    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269" podUID="717990dd-b98d-4baa-92da-ea34409707b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused"
Sep 16 18:28:51.080689 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:51.080640    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb" podUID="d90b9763-cd69-4879-9274-5f7f5ed9fcac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 16 18:28:51.437163 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:28:51.437050    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269" podUID="717990dd-b98d-4baa-92da-ea34409707b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused"
Sep 16 18:29:01.080556 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:29:01.080501    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb" podUID="d90b9763-cd69-4879-9274-5f7f5ed9fcac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 16 18:29:01.438074 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:29:01.437928    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269" podUID="717990dd-b98d-4baa-92da-ea34409707b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused"
Sep 16 18:29:11.081018 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:29:11.080952    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb" podUID="d90b9763-cd69-4879-9274-5f7f5ed9fcac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 16 18:29:11.437127 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:29:11.437018    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269" podUID="717990dd-b98d-4baa-92da-ea34409707b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused"
Sep 16 18:29:21.081108 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:29:21.081064    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb" podUID="d90b9763-cd69-4879-9274-5f7f5ed9fcac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 16 18:29:21.437541 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:29:21.437439    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269" podUID="717990dd-b98d-4baa-92da-ea34409707b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused"
Sep 16 18:29:31.080824 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:29:31.080779    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb" podUID="d90b9763-cd69-4879-9274-5f7f5ed9fcac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 16 18:29:31.438093 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:29:31.437963    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269" podUID="717990dd-b98d-4baa-92da-ea34409707b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused"
Sep 16 18:29:38.003765 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:29:38.003715    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:29:39.487007 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:29:39.486966    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:29:41.080566 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:29:41.080519    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb" podUID="d90b9763-cd69-4879-9274-5f7f5ed9fcac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 16 18:29:41.437896 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:29:41.437749    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269" podUID="717990dd-b98d-4baa-92da-ea34409707b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused"
Sep 16 18:29:51.081095 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:29:51.081051    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb" podUID="d90b9763-cd69-4879-9274-5f7f5ed9fcac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 16 18:29:51.437230 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:29:51.437131    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269" podUID="717990dd-b98d-4baa-92da-ea34409707b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused"
Sep 16 18:30:01.080492 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:01.080442    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb" podUID="d90b9763-cd69-4879-9274-5f7f5ed9fcac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 16 18:30:01.437848 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:01.437753    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269" podUID="717990dd-b98d-4baa-92da-ea34409707b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused"
Sep 16 18:30:11.081060 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:11.081008    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb" podUID="d90b9763-cd69-4879-9274-5f7f5ed9fcac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 16 18:30:11.437640 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:11.437526    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269" podUID="717990dd-b98d-4baa-92da-ea34409707b4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": dial tcp 10.133.0.86:8000: connect: connection refused"
Sep 16 18:30:21.081129 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:21.081083    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb" podUID="d90b9763-cd69-4879-9274-5f7f5ed9fcac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 16 18:30:21.447187 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:21.447104    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"
Sep 16 18:30:21.455146 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:21.455120    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"
Sep 16 18:30:29.345757 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:29.345718    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"]
Sep 16 18:30:29.348736 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:29.348701    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:30:29.412216 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:29.412172    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:30:31.081077 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:31.081031    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb" podUID="d90b9763-cd69-4879-9274-5f7f5ed9fcac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 16 18:30:41.080701 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:41.080652    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb" podUID="d90b9763-cd69-4879-9274-5f7f5ed9fcac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 16 18:30:44.370826 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:44.370788    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269" podUID="717990dd-b98d-4baa-92da-ea34409707b4" containerName="main" containerID="cri-o://4a7f38ef9bebb29c469a616c4410794f8817cf8ade9082f0c238546d60000921" gracePeriod=45
Sep 16 18:30:45.117290 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.117265    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"
Sep 16 18:30:45.183722 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.183641    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/717990dd-b98d-4baa-92da-ea34409707b4-kserve-provision-location\") pod \"717990dd-b98d-4baa-92da-ea34409707b4\" (UID: \"717990dd-b98d-4baa-92da-ea34409707b4\") "
Sep 16 18:30:45.183722 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.183680    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7x4r\" (UniqueName: \"kubernetes.io/projected/717990dd-b98d-4baa-92da-ea34409707b4-kube-api-access-q7x4r\") pod \"717990dd-b98d-4baa-92da-ea34409707b4\" (UID: \"717990dd-b98d-4baa-92da-ea34409707b4\") "
Sep 16 18:30:45.183722 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.183702    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/717990dd-b98d-4baa-92da-ea34409707b4-tls-certs\") pod \"717990dd-b98d-4baa-92da-ea34409707b4\" (UID: \"717990dd-b98d-4baa-92da-ea34409707b4\") "
Sep 16 18:30:45.184010 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.183724    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/717990dd-b98d-4baa-92da-ea34409707b4-home\") pod \"717990dd-b98d-4baa-92da-ea34409707b4\" (UID: \"717990dd-b98d-4baa-92da-ea34409707b4\") "
Sep 16 18:30:45.184010 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.183757    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/717990dd-b98d-4baa-92da-ea34409707b4-dshm\") pod \"717990dd-b98d-4baa-92da-ea34409707b4\" (UID: \"717990dd-b98d-4baa-92da-ea34409707b4\") "
Sep 16 18:30:45.184010 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.183785    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/717990dd-b98d-4baa-92da-ea34409707b4-model-cache\") pod \"717990dd-b98d-4baa-92da-ea34409707b4\" (UID: \"717990dd-b98d-4baa-92da-ea34409707b4\") "
Sep 16 18:30:45.184010 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.183805    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/717990dd-b98d-4baa-92da-ea34409707b4-tmp-dir\") pod \"717990dd-b98d-4baa-92da-ea34409707b4\" (UID: \"717990dd-b98d-4baa-92da-ea34409707b4\") "
Sep 16 18:30:45.184010 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.183855    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/717990dd-b98d-4baa-92da-ea34409707b4-cabundle-cert\") pod \"717990dd-b98d-4baa-92da-ea34409707b4\" (UID: \"717990dd-b98d-4baa-92da-ea34409707b4\") "
Sep 16 18:30:45.184283 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.184061    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/717990dd-b98d-4baa-92da-ea34409707b4-model-cache" (OuterVolumeSpecName: "model-cache") pod "717990dd-b98d-4baa-92da-ea34409707b4" (UID: "717990dd-b98d-4baa-92da-ea34409707b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:30:45.184341 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.184287    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/717990dd-b98d-4baa-92da-ea34409707b4-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:30:45.184575 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.184546    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/717990dd-b98d-4baa-92da-ea34409707b4-home" (OuterVolumeSpecName: "home") pod "717990dd-b98d-4baa-92da-ea34409707b4" (UID: "717990dd-b98d-4baa-92da-ea34409707b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:30:45.185224 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.185188    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/717990dd-b98d-4baa-92da-ea34409707b4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "717990dd-b98d-4baa-92da-ea34409707b4" (UID: "717990dd-b98d-4baa-92da-ea34409707b4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:30:45.186304 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.186275    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/717990dd-b98d-4baa-92da-ea34409707b4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "717990dd-b98d-4baa-92da-ea34409707b4" (UID: "717990dd-b98d-4baa-92da-ea34409707b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:30:45.186430 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.186413    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/717990dd-b98d-4baa-92da-ea34409707b4-kube-api-access-q7x4r" (OuterVolumeSpecName: "kube-api-access-q7x4r") pod "717990dd-b98d-4baa-92da-ea34409707b4" (UID: "717990dd-b98d-4baa-92da-ea34409707b4"). InnerVolumeSpecName "kube-api-access-q7x4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:30:45.186499 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.186452    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/717990dd-b98d-4baa-92da-ea34409707b4-dshm" (OuterVolumeSpecName: "dshm") pod "717990dd-b98d-4baa-92da-ea34409707b4" (UID: "717990dd-b98d-4baa-92da-ea34409707b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:30:45.189440 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.189418    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/717990dd-b98d-4baa-92da-ea34409707b4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "717990dd-b98d-4baa-92da-ea34409707b4" (UID: "717990dd-b98d-4baa-92da-ea34409707b4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:30:45.197208 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.197185    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/717990dd-b98d-4baa-92da-ea34409707b4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "717990dd-b98d-4baa-92da-ea34409707b4" (UID: "717990dd-b98d-4baa-92da-ea34409707b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:30:45.285691 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.285652    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/717990dd-b98d-4baa-92da-ea34409707b4-kserve-provision-location\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:30:45.285691 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.285685    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7x4r\" (UniqueName: \"kubernetes.io/projected/717990dd-b98d-4baa-92da-ea34409707b4-kube-api-access-q7x4r\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:30:45.285691 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.285696    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/717990dd-b98d-4baa-92da-ea34409707b4-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:30:45.285972 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.285713    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/717990dd-b98d-4baa-92da-ea34409707b4-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:30:45.285972 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.285721    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/717990dd-b98d-4baa-92da-ea34409707b4-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:30:45.285972 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.285729    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/717990dd-b98d-4baa-92da-ea34409707b4-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:30:45.285972 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.285739    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/717990dd-b98d-4baa-92da-ea34409707b4-cabundle-cert\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:30:45.445333 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.445237    2577 generic.go:356] "Generic (PLEG): container finished" podID="717990dd-b98d-4baa-92da-ea34409707b4" containerID="4a7f38ef9bebb29c469a616c4410794f8817cf8ade9082f0c238546d60000921" exitCode=0
Sep 16 18:30:45.445333 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.445293    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"
Sep 16 18:30:45.445333 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.445312    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269" event={"ID":"717990dd-b98d-4baa-92da-ea34409707b4","Type":"ContainerDied","Data":"4a7f38ef9bebb29c469a616c4410794f8817cf8ade9082f0c238546d60000921"}
Sep 16 18:30:45.445846 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.445376    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269" event={"ID":"717990dd-b98d-4baa-92da-ea34409707b4","Type":"ContainerDied","Data":"ac43f66d16f5999abf1aa4dc2cb26e9e5a9586904e5e05e722bd81570908b96c"}
Sep 16 18:30:45.445846 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.445407    2577 scope.go:117] "RemoveContainer" containerID="4a7f38ef9bebb29c469a616c4410794f8817cf8ade9082f0c238546d60000921"
Sep 16 18:30:45.455696 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.455676    2577 scope.go:117] "RemoveContainer" containerID="23d35b83f816e3e57f1b0e35743cd8cd4df4276813bd73f1bd70a50b29aa43d5"
Sep 16 18:30:45.464489 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.464466    2577 scope.go:117] "RemoveContainer" containerID="4a7f38ef9bebb29c469a616c4410794f8817cf8ade9082f0c238546d60000921"
Sep 16 18:30:45.464762 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:30:45.464739    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a7f38ef9bebb29c469a616c4410794f8817cf8ade9082f0c238546d60000921\": container with ID starting with 4a7f38ef9bebb29c469a616c4410794f8817cf8ade9082f0c238546d60000921 not found: ID does not exist" containerID="4a7f38ef9bebb29c469a616c4410794f8817cf8ade9082f0c238546d60000921"
Sep 16 18:30:45.464802 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.464772    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a7f38ef9bebb29c469a616c4410794f8817cf8ade9082f0c238546d60000921"} err="failed to get container status \"4a7f38ef9bebb29c469a616c4410794f8817cf8ade9082f0c238546d60000921\": rpc error: code = NotFound desc = could not find container \"4a7f38ef9bebb29c469a616c4410794f8817cf8ade9082f0c238546d60000921\": container with ID starting with 4a7f38ef9bebb29c469a616c4410794f8817cf8ade9082f0c238546d60000921 not found: ID does not exist"
Sep 16 18:30:45.464802 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.464790    2577 scope.go:117] "RemoveContainer" containerID="23d35b83f816e3e57f1b0e35743cd8cd4df4276813bd73f1bd70a50b29aa43d5"
Sep 16 18:30:45.465046 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:30:45.465022    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23d35b83f816e3e57f1b0e35743cd8cd4df4276813bd73f1bd70a50b29aa43d5\": container with ID starting with 23d35b83f816e3e57f1b0e35743cd8cd4df4276813bd73f1bd70a50b29aa43d5 not found: ID does not exist" containerID="23d35b83f816e3e57f1b0e35743cd8cd4df4276813bd73f1bd70a50b29aa43d5"
Sep 16 18:30:45.465125 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.465053    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23d35b83f816e3e57f1b0e35743cd8cd4df4276813bd73f1bd70a50b29aa43d5"} err="failed to get container status \"23d35b83f816e3e57f1b0e35743cd8cd4df4276813bd73f1bd70a50b29aa43d5\": rpc error: code = NotFound desc = could not find container \"23d35b83f816e3e57f1b0e35743cd8cd4df4276813bd73f1bd70a50b29aa43d5\": container with ID starting with 23d35b83f816e3e57f1b0e35743cd8cd4df4276813bd73f1bd70a50b29aa43d5 not found: ID does not exist"
Sep 16 18:30:45.470504 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.470478    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"]
Sep 16 18:30:45.476166 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:45.476146    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-7cbb94df-jm269"]
Sep 16 18:30:47.081664 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:47.081634    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="717990dd-b98d-4baa-92da-ea34409707b4" path="/var/lib/kubelet/pods/717990dd-b98d-4baa-92da-ea34409707b4/volumes"
Sep 16 18:30:51.092824 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:51.092797    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"
Sep 16 18:30:51.100461 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:51.100435    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"
Sep 16 18:30:52.887553 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:52.887514    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"]
Sep 16 18:30:52.895596 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:52.895561    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:30:53.037117 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:30:53.037068    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:31:07.910260 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:07.910212    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb" podUID="d90b9763-cd69-4879-9274-5f7f5ed9fcac" containerName="main" containerID="cri-o://a7c7a29b11c5c731ffbba77f4af3baa47a962443ff22f96712a83b1dd6f570ff" gracePeriod=45
Sep 16 18:31:08.093584 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:08.093532    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb" podUID="d90b9763-cd69-4879-9274-5f7f5ed9fcac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 16 18:31:08.579255 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:08.579221    2577 generic.go:356] "Generic (PLEG): container finished" podID="d90b9763-cd69-4879-9274-5f7f5ed9fcac" containerID="a7c7a29b11c5c731ffbba77f4af3baa47a962443ff22f96712a83b1dd6f570ff" exitCode=0
Sep 16 18:31:08.579393 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:08.579294    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb" event={"ID":"d90b9763-cd69-4879-9274-5f7f5ed9fcac","Type":"ContainerDied","Data":"a7c7a29b11c5c731ffbba77f4af3baa47a962443ff22f96712a83b1dd6f570ff"}
Sep 16 18:31:08.669931 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:08.669898    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"
Sep 16 18:31:08.790121 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:08.790024    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d90b9763-cd69-4879-9274-5f7f5ed9fcac-tls-certs\") pod \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\" (UID: \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\") "
Sep 16 18:31:08.790121 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:08.790070    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d90b9763-cd69-4879-9274-5f7f5ed9fcac-cabundle-cert\") pod \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\" (UID: \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\") "
Sep 16 18:31:08.790121 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:08.790106    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d90b9763-cd69-4879-9274-5f7f5ed9fcac-kserve-provision-location\") pod \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\" (UID: \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\") "
Sep 16 18:31:08.790121 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:08.790123    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d90b9763-cd69-4879-9274-5f7f5ed9fcac-model-cache\") pod \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\" (UID: \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\") "
Sep 16 18:31:08.790469 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:08.790141    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d90b9763-cd69-4879-9274-5f7f5ed9fcac-home\") pod \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\" (UID: \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\") "
Sep 16 18:31:08.790469 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:08.790169    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnjpk\" (UniqueName: \"kubernetes.io/projected/d90b9763-cd69-4879-9274-5f7f5ed9fcac-kube-api-access-vnjpk\") pod \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\" (UID: \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\") "
Sep 16 18:31:08.790469 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:08.790219    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d90b9763-cd69-4879-9274-5f7f5ed9fcac-dshm\") pod \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\" (UID: \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\") "
Sep 16 18:31:08.790469 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:08.790266    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d90b9763-cd69-4879-9274-5f7f5ed9fcac-tmp-dir\") pod \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\" (UID: \"d90b9763-cd69-4879-9274-5f7f5ed9fcac\") "
Sep 16 18:31:08.790469 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:08.790434    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90b9763-cd69-4879-9274-5f7f5ed9fcac-model-cache" (OuterVolumeSpecName: "model-cache") pod "d90b9763-cd69-4879-9274-5f7f5ed9fcac" (UID: "d90b9763-cd69-4879-9274-5f7f5ed9fcac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:31:08.790722 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:08.790536    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d90b9763-cd69-4879-9274-5f7f5ed9fcac-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d90b9763-cd69-4879-9274-5f7f5ed9fcac" (UID: "d90b9763-cd69-4879-9274-5f7f5ed9fcac"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:31:08.790722 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:08.790557    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d90b9763-cd69-4879-9274-5f7f5ed9fcac-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:31:08.790993 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:08.790967    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90b9763-cd69-4879-9274-5f7f5ed9fcac-home" (OuterVolumeSpecName: "home") pod "d90b9763-cd69-4879-9274-5f7f5ed9fcac" (UID: "d90b9763-cd69-4879-9274-5f7f5ed9fcac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:31:08.792667 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:08.792634    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90b9763-cd69-4879-9274-5f7f5ed9fcac-dshm" (OuterVolumeSpecName: "dshm") pod "d90b9763-cd69-4879-9274-5f7f5ed9fcac" (UID: "d90b9763-cd69-4879-9274-5f7f5ed9fcac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:31:08.792767 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:08.792698    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d90b9763-cd69-4879-9274-5f7f5ed9fcac-kube-api-access-vnjpk" (OuterVolumeSpecName: "kube-api-access-vnjpk") pod "d90b9763-cd69-4879-9274-5f7f5ed9fcac" (UID: "d90b9763-cd69-4879-9274-5f7f5ed9fcac"). InnerVolumeSpecName "kube-api-access-vnjpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:31:08.792837 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:08.792762    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d90b9763-cd69-4879-9274-5f7f5ed9fcac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d90b9763-cd69-4879-9274-5f7f5ed9fcac" (UID: "d90b9763-cd69-4879-9274-5f7f5ed9fcac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:31:08.800961 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:08.800935    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90b9763-cd69-4879-9274-5f7f5ed9fcac-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d90b9763-cd69-4879-9274-5f7f5ed9fcac" (UID: "d90b9763-cd69-4879-9274-5f7f5ed9fcac"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:31:08.804233 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:08.804208    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90b9763-cd69-4879-9274-5f7f5ed9fcac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d90b9763-cd69-4879-9274-5f7f5ed9fcac" (UID: "d90b9763-cd69-4879-9274-5f7f5ed9fcac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:31:08.891839 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:08.891793    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d90b9763-cd69-4879-9274-5f7f5ed9fcac-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:31:08.891839 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:08.891829    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d90b9763-cd69-4879-9274-5f7f5ed9fcac-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:31:08.891839 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:08.891845    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d90b9763-cd69-4879-9274-5f7f5ed9fcac-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:31:08.892109 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:08.891857    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d90b9763-cd69-4879-9274-5f7f5ed9fcac-cabundle-cert\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:31:08.892109 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:08.891896    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d90b9763-cd69-4879-9274-5f7f5ed9fcac-kserve-provision-location\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:31:08.892109 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:08.891909    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d90b9763-cd69-4879-9274-5f7f5ed9fcac-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:31:08.892109 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:08.891921    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnjpk\" (UniqueName: \"kubernetes.io/projected/d90b9763-cd69-4879-9274-5f7f5ed9fcac-kube-api-access-vnjpk\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:31:09.585707 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:09.585674    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"
Sep 16 18:31:09.586180 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:09.585710    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb" event={"ID":"d90b9763-cd69-4879-9274-5f7f5ed9fcac","Type":"ContainerDied","Data":"58ec7b3d3d974f7e8b9a7433905b9264d279b26465c097fd18af492cb51712dc"}
Sep 16 18:31:09.586180 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:09.585775    2577 scope.go:117] "RemoveContainer" containerID="a7c7a29b11c5c731ffbba77f4af3baa47a962443ff22f96712a83b1dd6f570ff"
Sep 16 18:31:09.595078 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:09.595060    2577 scope.go:117] "RemoveContainer" containerID="a4f842b93706ca9c04de4ce7d69f129fda68b9346a7e6027f935f818ca5f7c6a"
Sep 16 18:31:09.609211 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:09.609185    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"]
Sep 16 18:31:09.616974 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:09.616949    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-s9hcb"]
Sep 16 18:31:11.081653 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:11.081614    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d90b9763-cd69-4879-9274-5f7f5ed9fcac" path="/var/lib/kubelet/pods/d90b9763-cd69-4879-9274-5f7f5ed9fcac/volumes"
Sep 16 18:31:19.491561 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:19.491523    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:31:19.588420 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:19.588384    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:31:24.434283 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:24.434240    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"]
Sep 16 18:31:24.434772 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:24.434538    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6" podUID="daf3fcfd-2a37-4fef-b2f1-1299f26ddc86" containerName="istio-proxy" containerID="cri-o://f0995ebb10dd0639e68bd6625da78790b41abdf9504ce48ee8231ad9be325fd6" gracePeriod=30
Sep 16 18:31:24.528046 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:24.528012    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:31:29.687004 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.686978    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:31:29.690416 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.690386    2577 generic.go:356] "Generic (PLEG): container finished" podID="daf3fcfd-2a37-4fef-b2f1-1299f26ddc86" containerID="f0995ebb10dd0639e68bd6625da78790b41abdf9504ce48ee8231ad9be325fd6" exitCode=0
Sep 16 18:31:29.690522 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.690427    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"
Sep 16 18:31:29.690522 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.690475    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6" event={"ID":"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86","Type":"ContainerDied","Data":"f0995ebb10dd0639e68bd6625da78790b41abdf9504ce48ee8231ad9be325fd6"}
Sep 16 18:31:29.690605 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.690528    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6" event={"ID":"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86","Type":"ContainerDied","Data":"f820fe4ea4168c3e6cb4d17d888a73c8776c9f16eedad32ff5b978ecfd224fa8"}
Sep 16 18:31:29.690605 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.690557    2577 scope.go:117] "RemoveContainer" containerID="f0995ebb10dd0639e68bd6625da78790b41abdf9504ce48ee8231ad9be325fd6"
Sep 16 18:31:29.699014 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.698995    2577 scope.go:117] "RemoveContainer" containerID="51f62b4bfc5027301af942694661f709fc4752980bb0596e4a4183c84e4d0583"
Sep 16 18:31:29.707680 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.707658    2577 scope.go:117] "RemoveContainer" containerID="f0995ebb10dd0639e68bd6625da78790b41abdf9504ce48ee8231ad9be325fd6"
Sep 16 18:31:29.707986 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:31:29.707968    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0995ebb10dd0639e68bd6625da78790b41abdf9504ce48ee8231ad9be325fd6\": container with ID starting with f0995ebb10dd0639e68bd6625da78790b41abdf9504ce48ee8231ad9be325fd6 not found: ID does not exist" containerID="f0995ebb10dd0639e68bd6625da78790b41abdf9504ce48ee8231ad9be325fd6"
Sep 16 18:31:29.708077 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.707998    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0995ebb10dd0639e68bd6625da78790b41abdf9504ce48ee8231ad9be325fd6"} err="failed to get container status \"f0995ebb10dd0639e68bd6625da78790b41abdf9504ce48ee8231ad9be325fd6\": rpc error: code = NotFound desc = could not find container \"f0995ebb10dd0639e68bd6625da78790b41abdf9504ce48ee8231ad9be325fd6\": container with ID starting with f0995ebb10dd0639e68bd6625da78790b41abdf9504ce48ee8231ad9be325fd6 not found: ID does not exist"
Sep 16 18:31:29.708077 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.708021    2577 scope.go:117] "RemoveContainer" containerID="51f62b4bfc5027301af942694661f709fc4752980bb0596e4a4183c84e4d0583"
Sep 16 18:31:29.708299 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:31:29.708281    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51f62b4bfc5027301af942694661f709fc4752980bb0596e4a4183c84e4d0583\": container with ID starting with 51f62b4bfc5027301af942694661f709fc4752980bb0596e4a4183c84e4d0583 not found: ID does not exist" containerID="51f62b4bfc5027301af942694661f709fc4752980bb0596e4a4183c84e4d0583"
Sep 16 18:31:29.708343 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.708308    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51f62b4bfc5027301af942694661f709fc4752980bb0596e4a4183c84e4d0583"} err="failed to get container status \"51f62b4bfc5027301af942694661f709fc4752980bb0596e4a4183c84e4d0583\": rpc error: code = NotFound desc = could not find container \"51f62b4bfc5027301af942694661f709fc4752980bb0596e4a4183c84e4d0583\": container with ID starting with 51f62b4bfc5027301af942694661f709fc4752980bb0596e4a4183c84e4d0583 not found: ID does not exist"
Sep 16 18:31:29.879098 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.879065    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-istio-podinfo\") pod \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") "
Sep 16 18:31:29.879267 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.879128    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-credential-socket\") pod \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") "
Sep 16 18:31:29.879267 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.879171    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-workload-certs\") pod \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") "
Sep 16 18:31:29.879267 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.879198    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-istio-envoy\") pod \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") "
Sep 16 18:31:29.879267 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.879227    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-istiod-ca-cert\") pod \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") "
Sep 16 18:31:29.879571 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.879271    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-workload-socket\") pod \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") "
Sep 16 18:31:29.879571 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.879312    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-istio-data\") pod \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") "
Sep 16 18:31:29.879571 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.879342    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-istio-token\") pod \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") "
Sep 16 18:31:29.879571 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.879379    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62p8b\" (UniqueName: \"kubernetes.io/projected/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-kube-api-access-62p8b\") pod \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\" (UID: \"daf3fcfd-2a37-4fef-b2f1-1299f26ddc86\") "
Sep 16 18:31:29.879571 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.879384    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "daf3fcfd-2a37-4fef-b2f1-1299f26ddc86" (UID: "daf3fcfd-2a37-4fef-b2f1-1299f26ddc86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:31:29.879817 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.879618    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-credential-socket\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:31:29.879817 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.879637    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-istio-data" (OuterVolumeSpecName: "istio-data") pod "daf3fcfd-2a37-4fef-b2f1-1299f26ddc86" (UID: "daf3fcfd-2a37-4fef-b2f1-1299f26ddc86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:31:29.879817 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.879648    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "daf3fcfd-2a37-4fef-b2f1-1299f26ddc86" (UID: "daf3fcfd-2a37-4fef-b2f1-1299f26ddc86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:31:29.879817 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.879686    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "daf3fcfd-2a37-4fef-b2f1-1299f26ddc86" (UID: "daf3fcfd-2a37-4fef-b2f1-1299f26ddc86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:31:29.879817 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.879777    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "daf3fcfd-2a37-4fef-b2f1-1299f26ddc86" (UID: "daf3fcfd-2a37-4fef-b2f1-1299f26ddc86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:31:29.881565 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.881536    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "daf3fcfd-2a37-4fef-b2f1-1299f26ddc86" (UID: "daf3fcfd-2a37-4fef-b2f1-1299f26ddc86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:31:29.881685 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.881606    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-kube-api-access-62p8b" (OuterVolumeSpecName: "kube-api-access-62p8b") pod "daf3fcfd-2a37-4fef-b2f1-1299f26ddc86" (UID: "daf3fcfd-2a37-4fef-b2f1-1299f26ddc86"). InnerVolumeSpecName "kube-api-access-62p8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:31:29.881685 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.881625    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-istio-token" (OuterVolumeSpecName: "istio-token") pod "daf3fcfd-2a37-4fef-b2f1-1299f26ddc86" (UID: "daf3fcfd-2a37-4fef-b2f1-1299f26ddc86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:31:29.881685 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.881649    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "daf3fcfd-2a37-4fef-b2f1-1299f26ddc86" (UID: "daf3fcfd-2a37-4fef-b2f1-1299f26ddc86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:31:29.981097 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.980998    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-workload-socket\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:31:29.981097 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.981039    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-istio-data\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:31:29.981097 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.981051    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-istio-token\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:31:29.981097 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.981060    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62p8b\" (UniqueName: \"kubernetes.io/projected/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-kube-api-access-62p8b\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:31:29.981097 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.981070    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-istio-podinfo\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:31:29.981097 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.981078    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-workload-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:31:29.981097 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.981087    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-istio-envoy\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:31:29.981097 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:29.981097    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86-istiod-ca-cert\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:31:30.014436 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:30.014403    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"]
Sep 16 18:31:30.017792 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:30.017763    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-56bd58cf4d-p29n6"]
Sep 16 18:31:31.080792 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:31.080755    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="daf3fcfd-2a37-4fef-b2f1-1299f26ddc86" path="/var/lib/kubelet/pods/daf3fcfd-2a37-4fef-b2f1-1299f26ddc86/volumes"
Sep 16 18:31:42.663828 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:42.663785    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:31:42.764400 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:42.764360    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:31:43.886957 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.886917    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"]
Sep 16 18:31:43.887336 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.887253    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="717990dd-b98d-4baa-92da-ea34409707b4" containerName="main"
Sep 16 18:31:43.887336 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.887265    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="717990dd-b98d-4baa-92da-ea34409707b4" containerName="main"
Sep 16 18:31:43.887336 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.887272    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d90b9763-cd69-4879-9274-5f7f5ed9fcac" containerName="storage-initializer"
Sep 16 18:31:43.887336 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.887278    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d90b9763-cd69-4879-9274-5f7f5ed9fcac" containerName="storage-initializer"
Sep 16 18:31:43.887336 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.887292    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="717990dd-b98d-4baa-92da-ea34409707b4" containerName="storage-initializer"
Sep 16 18:31:43.887336 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.887298    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="717990dd-b98d-4baa-92da-ea34409707b4" containerName="storage-initializer"
Sep 16 18:31:43.887336 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.887313    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d90b9763-cd69-4879-9274-5f7f5ed9fcac" containerName="main"
Sep 16 18:31:43.887336 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.887318    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d90b9763-cd69-4879-9274-5f7f5ed9fcac" containerName="main"
Sep 16 18:31:43.887336 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.887325    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daf3fcfd-2a37-4fef-b2f1-1299f26ddc86" containerName="istio-proxy"
Sep 16 18:31:43.887336 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.887330    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="daf3fcfd-2a37-4fef-b2f1-1299f26ddc86" containerName="istio-proxy"
Sep 16 18:31:43.887336 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.887337    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daf3fcfd-2a37-4fef-b2f1-1299f26ddc86" containerName="istio-proxy"
Sep 16 18:31:43.887336 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.887343    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="daf3fcfd-2a37-4fef-b2f1-1299f26ddc86" containerName="istio-proxy"
Sep 16 18:31:43.887706 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.887396    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="717990dd-b98d-4baa-92da-ea34409707b4" containerName="main"
Sep 16 18:31:43.887706 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.887406    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d90b9763-cd69-4879-9274-5f7f5ed9fcac" containerName="main"
Sep 16 18:31:43.887706 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.887413    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="daf3fcfd-2a37-4fef-b2f1-1299f26ddc86" containerName="istio-proxy"
Sep 16 18:31:43.887706 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.887419    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="daf3fcfd-2a37-4fef-b2f1-1299f26ddc86" containerName="istio-proxy"
Sep 16 18:31:43.892265 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.892239    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:31:43.897277 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.897247    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-bxrqw\""
Sep 16 18:31:43.898051 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.898022    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-zrgww\""
Sep 16 18:31:43.898051 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.898040    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 16 18:31:43.898207 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.898025    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 16 18:31:43.898207 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.898024    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 16 18:31:43.898207 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.898069    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:31:43.898207 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.898027    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 16 18:31:43.910138 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.910112    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"]
Sep 16 18:31:43.981168 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.981135    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a9aae77-6433-4601-bdaa-88abe763dee7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk\" (UID: \"8a9aae77-6433-4601-bdaa-88abe763dee7\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:31:43.981353 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.981185    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a9aae77-6433-4601-bdaa-88abe763dee7-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk\" (UID: \"8a9aae77-6433-4601-bdaa-88abe763dee7\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:31:43.981353 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.981214    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a9aae77-6433-4601-bdaa-88abe763dee7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk\" (UID: \"8a9aae77-6433-4601-bdaa-88abe763dee7\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:31:43.981353 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.981238    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a9aae77-6433-4601-bdaa-88abe763dee7-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk\" (UID: \"8a9aae77-6433-4601-bdaa-88abe763dee7\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:31:43.981353 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.981276    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a9aae77-6433-4601-bdaa-88abe763dee7-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk\" (UID: \"8a9aae77-6433-4601-bdaa-88abe763dee7\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:31:43.981353 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.981291    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a9aae77-6433-4601-bdaa-88abe763dee7-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk\" (UID: \"8a9aae77-6433-4601-bdaa-88abe763dee7\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:31:43.981353 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.981331    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a9aae77-6433-4601-bdaa-88abe763dee7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk\" (UID: \"8a9aae77-6433-4601-bdaa-88abe763dee7\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:31:43.981353 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:43.981352    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsg4b\" (UniqueName: \"kubernetes.io/projected/8a9aae77-6433-4601-bdaa-88abe763dee7-kube-api-access-lsg4b\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk\" (UID: \"8a9aae77-6433-4601-bdaa-88abe763dee7\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:31:44.016802 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:44.016771    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:31:44.081965 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:44.081928    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a9aae77-6433-4601-bdaa-88abe763dee7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk\" (UID: \"8a9aae77-6433-4601-bdaa-88abe763dee7\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:31:44.081965 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:44.081967    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a9aae77-6433-4601-bdaa-88abe763dee7-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk\" (UID: \"8a9aae77-6433-4601-bdaa-88abe763dee7\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:31:44.082216 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:44.082150    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a9aae77-6433-4601-bdaa-88abe763dee7-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk\" (UID: \"8a9aae77-6433-4601-bdaa-88abe763dee7\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:31:44.082216 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:44.082188    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a9aae77-6433-4601-bdaa-88abe763dee7-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk\" (UID: \"8a9aae77-6433-4601-bdaa-88abe763dee7\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:31:44.082349 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:44.082225    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a9aae77-6433-4601-bdaa-88abe763dee7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk\" (UID: \"8a9aae77-6433-4601-bdaa-88abe763dee7\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:31:44.082349 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:44.082260    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsg4b\" (UniqueName: \"kubernetes.io/projected/8a9aae77-6433-4601-bdaa-88abe763dee7-kube-api-access-lsg4b\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk\" (UID: \"8a9aae77-6433-4601-bdaa-88abe763dee7\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:31:44.082349 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:44.082308    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a9aae77-6433-4601-bdaa-88abe763dee7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk\" (UID: \"8a9aae77-6433-4601-bdaa-88abe763dee7\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:31:44.082505 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:44.082384    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a9aae77-6433-4601-bdaa-88abe763dee7-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk\" (UID: \"8a9aae77-6433-4601-bdaa-88abe763dee7\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:31:44.082505 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:44.082422    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a9aae77-6433-4601-bdaa-88abe763dee7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk\" (UID: \"8a9aae77-6433-4601-bdaa-88abe763dee7\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:31:44.082505 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:44.082476    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a9aae77-6433-4601-bdaa-88abe763dee7-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk\" (UID: \"8a9aae77-6433-4601-bdaa-88abe763dee7\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:31:44.082670 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:44.082582    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a9aae77-6433-4601-bdaa-88abe763dee7-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk\" (UID: \"8a9aae77-6433-4601-bdaa-88abe763dee7\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:31:44.082759 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:44.082738    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a9aae77-6433-4601-bdaa-88abe763dee7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk\" (UID: \"8a9aae77-6433-4601-bdaa-88abe763dee7\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:31:44.082827 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:44.082738    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a9aae77-6433-4601-bdaa-88abe763dee7-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk\" (UID: \"8a9aae77-6433-4601-bdaa-88abe763dee7\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:31:44.084654 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:44.084630    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a9aae77-6433-4601-bdaa-88abe763dee7-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk\" (UID: \"8a9aae77-6433-4601-bdaa-88abe763dee7\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:31:44.084761 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:44.084744    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a9aae77-6433-4601-bdaa-88abe763dee7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk\" (UID: \"8a9aae77-6433-4601-bdaa-88abe763dee7\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:31:44.091672 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:44.091645    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsg4b\" (UniqueName: \"kubernetes.io/projected/8a9aae77-6433-4601-bdaa-88abe763dee7-kube-api-access-lsg4b\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk\" (UID: \"8a9aae77-6433-4601-bdaa-88abe763dee7\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:31:44.201504 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:44.201413    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:31:44.482573 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:44.482539    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"]
Sep 16 18:31:44.484045 ip-10-0-136-122 kubenswrapper[2577]: W0916 18:31:44.484016    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a9aae77_6433_4601_bdaa_88abe763dee7.slice/crio-85a8d35be29b880ca95c315d8c2aa7b04c8dfb5f7fd147424d693fa074995406 WatchSource:0}: Error finding container 85a8d35be29b880ca95c315d8c2aa7b04c8dfb5f7fd147424d693fa074995406: Status 404 returned error can't find the container with id 85a8d35be29b880ca95c315d8c2aa7b04c8dfb5f7fd147424d693fa074995406
Sep 16 18:31:44.656446 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:44.656414    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:31:44.770719 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:44.770634    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk" event={"ID":"8a9aae77-6433-4601-bdaa-88abe763dee7","Type":"ContainerStarted","Data":"0e000c223f00245195ed22c76bd1cccfe484003e672ed0e6430e332545512e15"}
Sep 16 18:31:44.770719 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:44.770686    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk" event={"ID":"8a9aae77-6433-4601-bdaa-88abe763dee7","Type":"ContainerStarted","Data":"85a8d35be29b880ca95c315d8c2aa7b04c8dfb5f7fd147424d693fa074995406"}
Sep 16 18:31:44.770940 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:44.770761    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:31:45.777929 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:45.777881    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk" event={"ID":"8a9aae77-6433-4601-bdaa-88abe763dee7","Type":"ContainerStarted","Data":"ab23fffced3d673a125f562e4a2801d48ed598312b1b351a0f51c60f90fb69b9"}
Sep 16 18:31:46.046389 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:46.046292    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:31:47.794963 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:47.794838    2577 generic.go:356] "Generic (PLEG): container finished" podID="8a9aae77-6433-4601-bdaa-88abe763dee7" containerID="ab23fffced3d673a125f562e4a2801d48ed598312b1b351a0f51c60f90fb69b9" exitCode=0
Sep 16 18:31:47.794963 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:47.794905    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk" event={"ID":"8a9aae77-6433-4601-bdaa-88abe763dee7","Type":"ContainerDied","Data":"ab23fffced3d673a125f562e4a2801d48ed598312b1b351a0f51c60f90fb69b9"}
Sep 16 18:31:48.803177 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:48.803138    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk" event={"ID":"8a9aae77-6433-4601-bdaa-88abe763dee7","Type":"ContainerStarted","Data":"a0f2d27fceaefe70986a9d25846daff06162812abb1f2868af82344007362d88"}
Sep 16 18:31:48.839791 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:48.839724    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk" podStartSLOduration=3.512353172 podStartE2EDuration="5.839700944s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.327347772s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:31:48.836584214 +0000 UTC m=+4030.199263013" watchObservedRunningTime="2026-09-16 18:31:48.839700944 +0000 UTC m=+4030.202379733"
Sep 16 18:31:49.214395 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:49.214361    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:31:49.794425 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:49.794391    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:31:54.202020 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:54.201972    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:31:54.202616 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:54.202061    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:31:54.203660 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:54.203380    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk" podUID="8a9aae77-6433-4601-bdaa-88abe763dee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8001/health\": dial tcp 10.133.0.88:8001: connect: connection refused"
Sep 16 18:31:54.215390 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:54.215359    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:31:55.436207 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:55.436156    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:31:55.976712 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:55.976668    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:31:56.057374 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:31:56.057332    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:32:03.582389 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.582309    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"]
Sep 16 18:32:03.586265 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.586242    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"
Sep 16 18:32:03.588976 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.588953    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Sep 16 18:32:03.589164 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.588994    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:32:03.589520 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.589497    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Sep 16 18:32:03.589639 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.589597    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-tc5jw\""
Sep 16 18:32:03.590224 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.590201    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Sep 16 18:32:03.590335 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.590256    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Sep 16 18:32:03.604081 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.604056    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"]
Sep 16 18:32:03.617021 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.616996    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:32:03.656771 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.656723    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6faad72-a48b-4dbb-9b7f-67226a813f4f-dshm\") pod \"lora-single-adapter-test-kserve-75ff599774-8r8c7\" (UID: \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"
Sep 16 18:32:03.656771 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.656777    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6faad72-a48b-4dbb-9b7f-67226a813f4f-tmp-dir\") pod \"lora-single-adapter-test-kserve-75ff599774-8r8c7\" (UID: \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"
Sep 16 18:32:03.657042 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.656830    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tczmb\" (UniqueName: \"kubernetes.io/projected/b6faad72-a48b-4dbb-9b7f-67226a813f4f-kube-api-access-tczmb\") pod \"lora-single-adapter-test-kserve-75ff599774-8r8c7\" (UID: \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"
Sep 16 18:32:03.657042 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.656856    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6faad72-a48b-4dbb-9b7f-67226a813f4f-home\") pod \"lora-single-adapter-test-kserve-75ff599774-8r8c7\" (UID: \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"
Sep 16 18:32:03.657042 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.656929    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6faad72-a48b-4dbb-9b7f-67226a813f4f-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-75ff599774-8r8c7\" (UID: \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"
Sep 16 18:32:03.657160 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.657039    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6faad72-a48b-4dbb-9b7f-67226a813f4f-model-cache\") pod \"lora-single-adapter-test-kserve-75ff599774-8r8c7\" (UID: \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"
Sep 16 18:32:03.657160 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.657069    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6faad72-a48b-4dbb-9b7f-67226a813f4f-cabundle-cert\") pod \"lora-single-adapter-test-kserve-75ff599774-8r8c7\" (UID: \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"
Sep 16 18:32:03.657160 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.657093    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6faad72-a48b-4dbb-9b7f-67226a813f4f-tls-certs\") pod \"lora-single-adapter-test-kserve-75ff599774-8r8c7\" (UID: \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"
Sep 16 18:32:03.757736 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.757688    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6faad72-a48b-4dbb-9b7f-67226a813f4f-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-75ff599774-8r8c7\" (UID: \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"
Sep 16 18:32:03.757947 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.757765    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6faad72-a48b-4dbb-9b7f-67226a813f4f-model-cache\") pod \"lora-single-adapter-test-kserve-75ff599774-8r8c7\" (UID: \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"
Sep 16 18:32:03.757947 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.757794    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6faad72-a48b-4dbb-9b7f-67226a813f4f-cabundle-cert\") pod \"lora-single-adapter-test-kserve-75ff599774-8r8c7\" (UID: \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"
Sep 16 18:32:03.757947 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.757820    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6faad72-a48b-4dbb-9b7f-67226a813f4f-tls-certs\") pod \"lora-single-adapter-test-kserve-75ff599774-8r8c7\" (UID: \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"
Sep 16 18:32:03.757947 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.757883    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6faad72-a48b-4dbb-9b7f-67226a813f4f-dshm\") pod \"lora-single-adapter-test-kserve-75ff599774-8r8c7\" (UID: \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"
Sep 16 18:32:03.757947 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.757913    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6faad72-a48b-4dbb-9b7f-67226a813f4f-tmp-dir\") pod \"lora-single-adapter-test-kserve-75ff599774-8r8c7\" (UID: \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"
Sep 16 18:32:03.757947 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.757941    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tczmb\" (UniqueName: \"kubernetes.io/projected/b6faad72-a48b-4dbb-9b7f-67226a813f4f-kube-api-access-tczmb\") pod \"lora-single-adapter-test-kserve-75ff599774-8r8c7\" (UID: \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"
Sep 16 18:32:03.758280 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.757969    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6faad72-a48b-4dbb-9b7f-67226a813f4f-home\") pod \"lora-single-adapter-test-kserve-75ff599774-8r8c7\" (UID: \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"
Sep 16 18:32:03.758280 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.758186    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6faad72-a48b-4dbb-9b7f-67226a813f4f-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-75ff599774-8r8c7\" (UID: \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"
Sep 16 18:32:03.758280 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.758228    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6faad72-a48b-4dbb-9b7f-67226a813f4f-model-cache\") pod \"lora-single-adapter-test-kserve-75ff599774-8r8c7\" (UID: \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"
Sep 16 18:32:03.758413 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.758302    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6faad72-a48b-4dbb-9b7f-67226a813f4f-home\") pod \"lora-single-adapter-test-kserve-75ff599774-8r8c7\" (UID: \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"
Sep 16 18:32:03.758451 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.758439    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6faad72-a48b-4dbb-9b7f-67226a813f4f-tmp-dir\") pod \"lora-single-adapter-test-kserve-75ff599774-8r8c7\" (UID: \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"
Sep 16 18:32:03.758638 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.758614    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6faad72-a48b-4dbb-9b7f-67226a813f4f-cabundle-cert\") pod \"lora-single-adapter-test-kserve-75ff599774-8r8c7\" (UID: \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"
Sep 16 18:32:03.760325 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.760298    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6faad72-a48b-4dbb-9b7f-67226a813f4f-dshm\") pod \"lora-single-adapter-test-kserve-75ff599774-8r8c7\" (UID: \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"
Sep 16 18:32:03.760612 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.760597    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6faad72-a48b-4dbb-9b7f-67226a813f4f-tls-certs\") pod \"lora-single-adapter-test-kserve-75ff599774-8r8c7\" (UID: \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"
Sep 16 18:32:03.766999 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.766971    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tczmb\" (UniqueName: \"kubernetes.io/projected/b6faad72-a48b-4dbb-9b7f-67226a813f4f-kube-api-access-tczmb\") pod \"lora-single-adapter-test-kserve-75ff599774-8r8c7\" (UID: \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"
Sep 16 18:32:03.899498 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:03.899421    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"
Sep 16 18:32:04.195079 ip-10-0-136-122 kubenswrapper[2577]: W0916 18:32:04.195037    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6faad72_a48b_4dbb_9b7f_67226a813f4f.slice/crio-1402872c8142e32c22edecd0c40770fed3f7957be1e443925d5f23313726047f WatchSource:0}: Error finding container 1402872c8142e32c22edecd0c40770fed3f7957be1e443925d5f23313726047f: Status 404 returned error can't find the container with id 1402872c8142e32c22edecd0c40770fed3f7957be1e443925d5f23313726047f
Sep 16 18:32:04.196170 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:04.196137    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"]
Sep 16 18:32:04.202185 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:04.202054    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk" podUID="8a9aae77-6433-4601-bdaa-88abe763dee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8001/health\": dial tcp 10.133.0.88:8001: connect: connection refused"
Sep 16 18:32:04.897567 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:04.897528    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7" event={"ID":"b6faad72-a48b-4dbb-9b7f-67226a813f4f","Type":"ContainerStarted","Data":"1ee346174afb45b229aed1db9d87d47b32ed8af88b4fc0ee19c76c8bda411891"}
Sep 16 18:32:04.897992 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:04.897589    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7" event={"ID":"b6faad72-a48b-4dbb-9b7f-67226a813f4f","Type":"ContainerStarted","Data":"1402872c8142e32c22edecd0c40770fed3f7957be1e443925d5f23313726047f"}
Sep 16 18:32:07.918164 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:07.918125    2577 generic.go:356] "Generic (PLEG): container finished" podID="b6faad72-a48b-4dbb-9b7f-67226a813f4f" containerID="1ee346174afb45b229aed1db9d87d47b32ed8af88b4fc0ee19c76c8bda411891" exitCode=0
Sep 16 18:32:07.918552 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:07.918201    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7" event={"ID":"b6faad72-a48b-4dbb-9b7f-67226a813f4f","Type":"ContainerDied","Data":"1ee346174afb45b229aed1db9d87d47b32ed8af88b4fc0ee19c76c8bda411891"}
Sep 16 18:32:08.052658 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:08.052571    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:32:08.926581 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:08.926541    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7" event={"ID":"b6faad72-a48b-4dbb-9b7f-67226a813f4f","Type":"ContainerStarted","Data":"f7cebaf3ee4cfd4406c17c84bc9415991af7bad38790e45c4bed2a5c052f37b3"}
Sep 16 18:32:08.953680 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:08.953625    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7" podStartSLOduration=3.136067222 podStartE2EDuration="5.953607896s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.817540674s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:32:08.950153756 +0000 UTC m=+4050.312832565" watchObservedRunningTime="2026-09-16 18:32:08.953607896 +0000 UTC m=+4050.316286681"
Sep 16 18:32:09.900079 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:09.900027    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"
Sep 16 18:32:10.274831 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:10.274731    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:32:10.339905 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:10.339843    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:32:13.899820 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:13.899773    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"
Sep 16 18:32:13.901903 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:13.901847    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7" podUID="b6faad72-a48b-4dbb-9b7f-67226a813f4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": dial tcp 10.133.0.89:8000: connect: connection refused"
Sep 16 18:32:14.202542 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:14.202425    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk" podUID="8a9aae77-6433-4601-bdaa-88abe763dee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8001/health\": dial tcp 10.133.0.88:8001: connect: connection refused"
Sep 16 18:32:21.318247 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:21.318207    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:32:23.900464 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:23.900411    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7" podUID="b6faad72-a48b-4dbb-9b7f-67226a813f4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": dial tcp 10.133.0.89:8000: connect: connection refused"
Sep 16 18:32:24.202831 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:24.202708    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk" podUID="8a9aae77-6433-4601-bdaa-88abe763dee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8001/health\": dial tcp 10.133.0.88:8001: connect: connection refused"
Sep 16 18:32:33.900405 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:33.900350    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7" podUID="b6faad72-a48b-4dbb-9b7f-67226a813f4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": dial tcp 10.133.0.89:8000: connect: connection refused"
Sep 16 18:32:34.202405 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:34.202281    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk" podUID="8a9aae77-6433-4601-bdaa-88abe763dee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8001/health\": dial tcp 10.133.0.88:8001: connect: connection refused"
Sep 16 18:32:43.900594 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:43.900535    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7" podUID="b6faad72-a48b-4dbb-9b7f-67226a813f4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": dial tcp 10.133.0.89:8000: connect: connection refused"
Sep 16 18:32:44.202529 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:44.202423    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk" podUID="8a9aae77-6433-4601-bdaa-88abe763dee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8001/health\": dial tcp 10.133.0.88:8001: connect: connection refused"
Sep 16 18:32:53.900082 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:53.900033    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7" podUID="b6faad72-a48b-4dbb-9b7f-67226a813f4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": dial tcp 10.133.0.89:8000: connect: connection refused"
Sep 16 18:32:54.202161 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:32:54.202033    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk" podUID="8a9aae77-6433-4601-bdaa-88abe763dee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8001/health\": dial tcp 10.133.0.88:8001: connect: connection refused"
Sep 16 18:33:03.899904 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:33:03.899837    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7" podUID="b6faad72-a48b-4dbb-9b7f-67226a813f4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": dial tcp 10.133.0.89:8000: connect: connection refused"
Sep 16 18:33:04.202606 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:33:04.202483    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk" podUID="8a9aae77-6433-4601-bdaa-88abe763dee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8001/health\": dial tcp 10.133.0.88:8001: connect: connection refused"
Sep 16 18:33:13.900313 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:33:13.900258    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7" podUID="b6faad72-a48b-4dbb-9b7f-67226a813f4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": dial tcp 10.133.0.89:8000: connect: connection refused"
Sep 16 18:33:14.202201 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:33:14.202089    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk" podUID="8a9aae77-6433-4601-bdaa-88abe763dee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8001/health\": dial tcp 10.133.0.88:8001: connect: connection refused"
Sep 16 18:33:23.911188 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:33:23.911153    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"
Sep 16 18:33:23.919399 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:33:23.919358    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"
Sep 16 18:33:24.202242 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:33:24.202140    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk" podUID="8a9aae77-6433-4601-bdaa-88abe763dee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8001/health\": dial tcp 10.133.0.88:8001: connect: connection refused"
Sep 16 18:33:34.201947 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:33:34.201826    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk" podUID="8a9aae77-6433-4601-bdaa-88abe763dee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8001/health\": dial tcp 10.133.0.88:8001: connect: connection refused"
Sep 16 18:33:44.201829 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:33:44.201777    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk" podUID="8a9aae77-6433-4601-bdaa-88abe763dee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8001/health\": dial tcp 10.133.0.88:8001: connect: connection refused"
Sep 16 18:33:54.201997 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:33:54.201947    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk" podUID="8a9aae77-6433-4601-bdaa-88abe763dee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8001/health\": dial tcp 10.133.0.88:8001: connect: connection refused"
Sep 16 18:34:04.221388 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:04.221345    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:34:04.241232 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:04.241197    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:34:06.929627 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:06.929595    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-75ff599774-8r8c7_b6faad72-a48b-4dbb-9b7f-67226a813f4f/storage-initializer/0.log"
Sep 16 18:34:06.938408 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:06.938384    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-75ff599774-8r8c7_b6faad72-a48b-4dbb-9b7f-67226a813f4f/main/0.log"
Sep 16 18:34:08.359235 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:08.359196    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"]
Sep 16 18:34:08.386597 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:08.386568    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:14.603716 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:14.603685    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:14.638093 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:14.638055    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"]
Sep 16 18:34:14.674501 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:14.674467    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:23.382099 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:23.382036    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7" podUID="b6faad72-a48b-4dbb-9b7f-67226a813f4f" containerName="main" containerID="cri-o://f7cebaf3ee4cfd4406c17c84bc9415991af7bad38790e45c4bed2a5c052f37b3" gracePeriod=45
Sep 16 18:34:23.912282 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:23.912242    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7" podUID="b6faad72-a48b-4dbb-9b7f-67226a813f4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": dial tcp 10.133.0.89:8000: connect: connection refused"
Sep 16 18:34:24.039195 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.039164    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"
Sep 16 18:34:24.156350 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.156257    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6faad72-a48b-4dbb-9b7f-67226a813f4f-kserve-provision-location\") pod \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\" (UID: \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\") "
Sep 16 18:34:24.156350 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.156320    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6faad72-a48b-4dbb-9b7f-67226a813f4f-model-cache\") pod \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\" (UID: \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\") "
Sep 16 18:34:24.156350 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.156344    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6faad72-a48b-4dbb-9b7f-67226a813f4f-cabundle-cert\") pod \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\" (UID: \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\") "
Sep 16 18:34:24.156642 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.156367    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6faad72-a48b-4dbb-9b7f-67226a813f4f-tls-certs\") pod \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\" (UID: \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\") "
Sep 16 18:34:24.156642 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.156410    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6faad72-a48b-4dbb-9b7f-67226a813f4f-home\") pod \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\" (UID: \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\") "
Sep 16 18:34:24.156642 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.156574    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6faad72-a48b-4dbb-9b7f-67226a813f4f-tmp-dir\") pod \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\" (UID: \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\") "
Sep 16 18:34:24.156642 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.156629    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tczmb\" (UniqueName: \"kubernetes.io/projected/b6faad72-a48b-4dbb-9b7f-67226a813f4f-kube-api-access-tczmb\") pod \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\" (UID: \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\") "
Sep 16 18:34:24.156882 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.156661    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6faad72-a48b-4dbb-9b7f-67226a813f4f-dshm\") pod \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\" (UID: \"b6faad72-a48b-4dbb-9b7f-67226a813f4f\") "
Sep 16 18:34:24.156882 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.156678    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6faad72-a48b-4dbb-9b7f-67226a813f4f-model-cache" (OuterVolumeSpecName: "model-cache") pod "b6faad72-a48b-4dbb-9b7f-67226a813f4f" (UID: "b6faad72-a48b-4dbb-9b7f-67226a813f4f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:34:24.156882 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.156845    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6faad72-a48b-4dbb-9b7f-67226a813f4f-home" (OuterVolumeSpecName: "home") pod "b6faad72-a48b-4dbb-9b7f-67226a813f4f" (UID: "b6faad72-a48b-4dbb-9b7f-67226a813f4f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:34:24.157111 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.156853    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6faad72-a48b-4dbb-9b7f-67226a813f4f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b6faad72-a48b-4dbb-9b7f-67226a813f4f" (UID: "b6faad72-a48b-4dbb-9b7f-67226a813f4f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:34:24.157111 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.156997    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6faad72-a48b-4dbb-9b7f-67226a813f4f-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:34:24.157111 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.157016    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6faad72-a48b-4dbb-9b7f-67226a813f4f-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:34:24.157111 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.157031    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6faad72-a48b-4dbb-9b7f-67226a813f4f-cabundle-cert\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:34:24.159115 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.159085    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6faad72-a48b-4dbb-9b7f-67226a813f4f-dshm" (OuterVolumeSpecName: "dshm") pod "b6faad72-a48b-4dbb-9b7f-67226a813f4f" (UID: "b6faad72-a48b-4dbb-9b7f-67226a813f4f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:34:24.159226 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.159156    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6faad72-a48b-4dbb-9b7f-67226a813f4f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b6faad72-a48b-4dbb-9b7f-67226a813f4f" (UID: "b6faad72-a48b-4dbb-9b7f-67226a813f4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:34:24.159472 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.159452    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6faad72-a48b-4dbb-9b7f-67226a813f4f-kube-api-access-tczmb" (OuterVolumeSpecName: "kube-api-access-tczmb") pod "b6faad72-a48b-4dbb-9b7f-67226a813f4f" (UID: "b6faad72-a48b-4dbb-9b7f-67226a813f4f"). InnerVolumeSpecName "kube-api-access-tczmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:34:24.168777 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.168753    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6faad72-a48b-4dbb-9b7f-67226a813f4f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b6faad72-a48b-4dbb-9b7f-67226a813f4f" (UID: "b6faad72-a48b-4dbb-9b7f-67226a813f4f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:34:24.179050 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.179028    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6faad72-a48b-4dbb-9b7f-67226a813f4f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b6faad72-a48b-4dbb-9b7f-67226a813f4f" (UID: "b6faad72-a48b-4dbb-9b7f-67226a813f4f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:34:24.258419 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.258376    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6faad72-a48b-4dbb-9b7f-67226a813f4f-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:34:24.258419 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.258414    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tczmb\" (UniqueName: \"kubernetes.io/projected/b6faad72-a48b-4dbb-9b7f-67226a813f4f-kube-api-access-tczmb\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:34:24.258419 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.258423    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6faad72-a48b-4dbb-9b7f-67226a813f4f-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:34:24.258668 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.258436    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6faad72-a48b-4dbb-9b7f-67226a813f4f-kserve-provision-location\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:34:24.258668 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.258452    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6faad72-a48b-4dbb-9b7f-67226a813f4f-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:34:24.741389 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.741352    2577 generic.go:356] "Generic (PLEG): container finished" podID="b6faad72-a48b-4dbb-9b7f-67226a813f4f" containerID="f7cebaf3ee4cfd4406c17c84bc9415991af7bad38790e45c4bed2a5c052f37b3" exitCode=0
Sep 16 18:34:24.741824 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.741402    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"
Sep 16 18:34:24.741824 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.741427    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7" event={"ID":"b6faad72-a48b-4dbb-9b7f-67226a813f4f","Type":"ContainerDied","Data":"f7cebaf3ee4cfd4406c17c84bc9415991af7bad38790e45c4bed2a5c052f37b3"}
Sep 16 18:34:24.741824 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.741480    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7" event={"ID":"b6faad72-a48b-4dbb-9b7f-67226a813f4f","Type":"ContainerDied","Data":"1402872c8142e32c22edecd0c40770fed3f7957be1e443925d5f23313726047f"}
Sep 16 18:34:24.741824 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.741502    2577 scope.go:117] "RemoveContainer" containerID="f7cebaf3ee4cfd4406c17c84bc9415991af7bad38790e45c4bed2a5c052f37b3"
Sep 16 18:34:24.751983 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.751962    2577 scope.go:117] "RemoveContainer" containerID="1ee346174afb45b229aed1db9d87d47b32ed8af88b4fc0ee19c76c8bda411891"
Sep 16 18:34:24.760895 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.760876    2577 scope.go:117] "RemoveContainer" containerID="f7cebaf3ee4cfd4406c17c84bc9415991af7bad38790e45c4bed2a5c052f37b3"
Sep 16 18:34:24.761176 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:34:24.761156    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7cebaf3ee4cfd4406c17c84bc9415991af7bad38790e45c4bed2a5c052f37b3\": container with ID starting with f7cebaf3ee4cfd4406c17c84bc9415991af7bad38790e45c4bed2a5c052f37b3 not found: ID does not exist" containerID="f7cebaf3ee4cfd4406c17c84bc9415991af7bad38790e45c4bed2a5c052f37b3"
Sep 16 18:34:24.761248 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.761184    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7cebaf3ee4cfd4406c17c84bc9415991af7bad38790e45c4bed2a5c052f37b3"} err="failed to get container status \"f7cebaf3ee4cfd4406c17c84bc9415991af7bad38790e45c4bed2a5c052f37b3\": rpc error: code = NotFound desc = could not find container \"f7cebaf3ee4cfd4406c17c84bc9415991af7bad38790e45c4bed2a5c052f37b3\": container with ID starting with f7cebaf3ee4cfd4406c17c84bc9415991af7bad38790e45c4bed2a5c052f37b3 not found: ID does not exist"
Sep 16 18:34:24.761248 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.761202    2577 scope.go:117] "RemoveContainer" containerID="1ee346174afb45b229aed1db9d87d47b32ed8af88b4fc0ee19c76c8bda411891"
Sep 16 18:34:24.761442 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:34:24.761424    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ee346174afb45b229aed1db9d87d47b32ed8af88b4fc0ee19c76c8bda411891\": container with ID starting with 1ee346174afb45b229aed1db9d87d47b32ed8af88b4fc0ee19c76c8bda411891 not found: ID does not exist" containerID="1ee346174afb45b229aed1db9d87d47b32ed8af88b4fc0ee19c76c8bda411891"
Sep 16 18:34:24.761483 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.761452    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ee346174afb45b229aed1db9d87d47b32ed8af88b4fc0ee19c76c8bda411891"} err="failed to get container status \"1ee346174afb45b229aed1db9d87d47b32ed8af88b4fc0ee19c76c8bda411891\": rpc error: code = NotFound desc = could not find container \"1ee346174afb45b229aed1db9d87d47b32ed8af88b4fc0ee19c76c8bda411891\": container with ID starting with 1ee346174afb45b229aed1db9d87d47b32ed8af88b4fc0ee19c76c8bda411891 not found: ID does not exist"
Sep 16 18:34:24.771829 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.771806    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"]
Sep 16 18:34:24.775884 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:24.775847    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75ff599774-8r8c7"]
Sep 16 18:34:25.081085 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:25.081051    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6faad72-a48b-4dbb-9b7f-67226a813f4f" path="/var/lib/kubelet/pods/b6faad72-a48b-4dbb-9b7f-67226a813f4f/volumes"
Sep 16 18:34:27.342822 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:27.342782    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:28.695142 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:28.695105    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:28.812601 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:28.812559    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:28.912218 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:28.912184    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:29.661062 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:29.661025    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk" podUID="8a9aae77-6433-4601-bdaa-88abe763dee7" containerName="main" containerID="cri-o://a0f2d27fceaefe70986a9d25846daff06162812abb1f2868af82344007362d88" gracePeriod=45
Sep 16 18:34:29.739994 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:29.739957    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:30.279159 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.279126    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk" podUID="8a9aae77-6433-4601-bdaa-88abe763dee7" containerName="llm-d-routing-sidecar" containerID="cri-o://0e000c223f00245195ed22c76bd1cccfe484003e672ed0e6430e332545512e15" gracePeriod=45
Sep 16 18:34:30.541803 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.541774    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:34:30.615611 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.615576    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsg4b\" (UniqueName: \"kubernetes.io/projected/8a9aae77-6433-4601-bdaa-88abe763dee7-kube-api-access-lsg4b\") pod \"8a9aae77-6433-4601-bdaa-88abe763dee7\" (UID: \"8a9aae77-6433-4601-bdaa-88abe763dee7\") "
Sep 16 18:34:30.615778 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.615632    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a9aae77-6433-4601-bdaa-88abe763dee7-kserve-provision-location\") pod \"8a9aae77-6433-4601-bdaa-88abe763dee7\" (UID: \"8a9aae77-6433-4601-bdaa-88abe763dee7\") "
Sep 16 18:34:30.615778 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.615657    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a9aae77-6433-4601-bdaa-88abe763dee7-tmp-dir\") pod \"8a9aae77-6433-4601-bdaa-88abe763dee7\" (UID: \"8a9aae77-6433-4601-bdaa-88abe763dee7\") "
Sep 16 18:34:30.615778 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.615752    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a9aae77-6433-4601-bdaa-88abe763dee7-tls-certs\") pod \"8a9aae77-6433-4601-bdaa-88abe763dee7\" (UID: \"8a9aae77-6433-4601-bdaa-88abe763dee7\") "
Sep 16 18:34:30.615980 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.615820    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a9aae77-6433-4601-bdaa-88abe763dee7-model-cache\") pod \"8a9aae77-6433-4601-bdaa-88abe763dee7\" (UID: \"8a9aae77-6433-4601-bdaa-88abe763dee7\") "
Sep 16 18:34:30.615980 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.615900    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a9aae77-6433-4601-bdaa-88abe763dee7-dshm\") pod \"8a9aae77-6433-4601-bdaa-88abe763dee7\" (UID: \"8a9aae77-6433-4601-bdaa-88abe763dee7\") "
Sep 16 18:34:30.615980 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.615920    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a9aae77-6433-4601-bdaa-88abe763dee7-cabundle-cert\") pod \"8a9aae77-6433-4601-bdaa-88abe763dee7\" (UID: \"8a9aae77-6433-4601-bdaa-88abe763dee7\") "
Sep 16 18:34:30.615980 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.615943    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a9aae77-6433-4601-bdaa-88abe763dee7-home\") pod \"8a9aae77-6433-4601-bdaa-88abe763dee7\" (UID: \"8a9aae77-6433-4601-bdaa-88abe763dee7\") "
Sep 16 18:34:30.616187 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.616086    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9aae77-6433-4601-bdaa-88abe763dee7-model-cache" (OuterVolumeSpecName: "model-cache") pod "8a9aae77-6433-4601-bdaa-88abe763dee7" (UID: "8a9aae77-6433-4601-bdaa-88abe763dee7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:34:30.616297 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.616278    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a9aae77-6433-4601-bdaa-88abe763dee7-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:34:30.616442 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.616412    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a9aae77-6433-4601-bdaa-88abe763dee7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8a9aae77-6433-4601-bdaa-88abe763dee7" (UID: "8a9aae77-6433-4601-bdaa-88abe763dee7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:34:30.616720 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.616693    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9aae77-6433-4601-bdaa-88abe763dee7-home" (OuterVolumeSpecName: "home") pod "8a9aae77-6433-4601-bdaa-88abe763dee7" (UID: "8a9aae77-6433-4601-bdaa-88abe763dee7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:34:30.618232 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.618203    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9aae77-6433-4601-bdaa-88abe763dee7-dshm" (OuterVolumeSpecName: "dshm") pod "8a9aae77-6433-4601-bdaa-88abe763dee7" (UID: "8a9aae77-6433-4601-bdaa-88abe763dee7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:34:30.618331 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.618265    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a9aae77-6433-4601-bdaa-88abe763dee7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a9aae77-6433-4601-bdaa-88abe763dee7" (UID: "8a9aae77-6433-4601-bdaa-88abe763dee7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:34:30.618479 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.618456    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a9aae77-6433-4601-bdaa-88abe763dee7-kube-api-access-lsg4b" (OuterVolumeSpecName: "kube-api-access-lsg4b") pod "8a9aae77-6433-4601-bdaa-88abe763dee7" (UID: "8a9aae77-6433-4601-bdaa-88abe763dee7"). InnerVolumeSpecName "kube-api-access-lsg4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:34:30.632571 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.632542    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9aae77-6433-4601-bdaa-88abe763dee7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8a9aae77-6433-4601-bdaa-88abe763dee7" (UID: "8a9aae77-6433-4601-bdaa-88abe763dee7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:34:30.632800 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.632774    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9aae77-6433-4601-bdaa-88abe763dee7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8a9aae77-6433-4601-bdaa-88abe763dee7" (UID: "8a9aae77-6433-4601-bdaa-88abe763dee7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:34:30.717300 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.717199    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsg4b\" (UniqueName: \"kubernetes.io/projected/8a9aae77-6433-4601-bdaa-88abe763dee7-kube-api-access-lsg4b\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:34:30.717300 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.717231    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a9aae77-6433-4601-bdaa-88abe763dee7-kserve-provision-location\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:34:30.717300 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.717248    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a9aae77-6433-4601-bdaa-88abe763dee7-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:34:30.717300 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.717262    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a9aae77-6433-4601-bdaa-88abe763dee7-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:34:30.717300 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.717272    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a9aae77-6433-4601-bdaa-88abe763dee7-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:34:30.717300 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.717284    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a9aae77-6433-4601-bdaa-88abe763dee7-cabundle-cert\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:34:30.717300 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.717295    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a9aae77-6433-4601-bdaa-88abe763dee7-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:34:30.778058 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.778021    2577 generic.go:356] "Generic (PLEG): container finished" podID="8a9aae77-6433-4601-bdaa-88abe763dee7" containerID="a0f2d27fceaefe70986a9d25846daff06162812abb1f2868af82344007362d88" exitCode=0
Sep 16 18:34:30.778058 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.778049    2577 generic.go:356] "Generic (PLEG): container finished" podID="8a9aae77-6433-4601-bdaa-88abe763dee7" containerID="0e000c223f00245195ed22c76bd1cccfe484003e672ed0e6430e332545512e15" exitCode=0
Sep 16 18:34:30.778496 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.778105    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk" event={"ID":"8a9aae77-6433-4601-bdaa-88abe763dee7","Type":"ContainerDied","Data":"a0f2d27fceaefe70986a9d25846daff06162812abb1f2868af82344007362d88"}
Sep 16 18:34:30.778496 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.778126    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"
Sep 16 18:34:30.778496 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.778151    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk" event={"ID":"8a9aae77-6433-4601-bdaa-88abe763dee7","Type":"ContainerDied","Data":"0e000c223f00245195ed22c76bd1cccfe484003e672ed0e6430e332545512e15"}
Sep 16 18:34:30.778496 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.778175    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk" event={"ID":"8a9aae77-6433-4601-bdaa-88abe763dee7","Type":"ContainerDied","Data":"85a8d35be29b880ca95c315d8c2aa7b04c8dfb5f7fd147424d693fa074995406"}
Sep 16 18:34:30.778496 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.778197    2577 scope.go:117] "RemoveContainer" containerID="a0f2d27fceaefe70986a9d25846daff06162812abb1f2868af82344007362d88"
Sep 16 18:34:30.789459 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.789435    2577 scope.go:117] "RemoveContainer" containerID="ab23fffced3d673a125f562e4a2801d48ed598312b1b351a0f51c60f90fb69b9"
Sep 16 18:34:30.798078 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.798056    2577 scope.go:117] "RemoveContainer" containerID="0e000c223f00245195ed22c76bd1cccfe484003e672ed0e6430e332545512e15"
Sep 16 18:34:30.805844 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.805825    2577 scope.go:117] "RemoveContainer" containerID="a0f2d27fceaefe70986a9d25846daff06162812abb1f2868af82344007362d88"
Sep 16 18:34:30.806136 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:34:30.806115    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0f2d27fceaefe70986a9d25846daff06162812abb1f2868af82344007362d88\": container with ID starting with a0f2d27fceaefe70986a9d25846daff06162812abb1f2868af82344007362d88 not found: ID does not exist" containerID="a0f2d27fceaefe70986a9d25846daff06162812abb1f2868af82344007362d88"
Sep 16 18:34:30.806182 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.806146    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0f2d27fceaefe70986a9d25846daff06162812abb1f2868af82344007362d88"} err="failed to get container status \"a0f2d27fceaefe70986a9d25846daff06162812abb1f2868af82344007362d88\": rpc error: code = NotFound desc = could not find container \"a0f2d27fceaefe70986a9d25846daff06162812abb1f2868af82344007362d88\": container with ID starting with a0f2d27fceaefe70986a9d25846daff06162812abb1f2868af82344007362d88 not found: ID does not exist"
Sep 16 18:34:30.806182 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.806164    2577 scope.go:117] "RemoveContainer" containerID="ab23fffced3d673a125f562e4a2801d48ed598312b1b351a0f51c60f90fb69b9"
Sep 16 18:34:30.806410 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:34:30.806391    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab23fffced3d673a125f562e4a2801d48ed598312b1b351a0f51c60f90fb69b9\": container with ID starting with ab23fffced3d673a125f562e4a2801d48ed598312b1b351a0f51c60f90fb69b9 not found: ID does not exist" containerID="ab23fffced3d673a125f562e4a2801d48ed598312b1b351a0f51c60f90fb69b9"
Sep 16 18:34:30.806455 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.806415    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab23fffced3d673a125f562e4a2801d48ed598312b1b351a0f51c60f90fb69b9"} err="failed to get container status \"ab23fffced3d673a125f562e4a2801d48ed598312b1b351a0f51c60f90fb69b9\": rpc error: code = NotFound desc = could not find container \"ab23fffced3d673a125f562e4a2801d48ed598312b1b351a0f51c60f90fb69b9\": container with ID starting with ab23fffced3d673a125f562e4a2801d48ed598312b1b351a0f51c60f90fb69b9 not found: ID does not exist"
Sep 16 18:34:30.806455 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.806431    2577 scope.go:117] "RemoveContainer" containerID="0e000c223f00245195ed22c76bd1cccfe484003e672ed0e6430e332545512e15"
Sep 16 18:34:30.806653 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:34:30.806634    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e000c223f00245195ed22c76bd1cccfe484003e672ed0e6430e332545512e15\": container with ID starting with 0e000c223f00245195ed22c76bd1cccfe484003e672ed0e6430e332545512e15 not found: ID does not exist" containerID="0e000c223f00245195ed22c76bd1cccfe484003e672ed0e6430e332545512e15"
Sep 16 18:34:30.806696 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.806657    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e000c223f00245195ed22c76bd1cccfe484003e672ed0e6430e332545512e15"} err="failed to get container status \"0e000c223f00245195ed22c76bd1cccfe484003e672ed0e6430e332545512e15\": rpc error: code = NotFound desc = could not find container \"0e000c223f00245195ed22c76bd1cccfe484003e672ed0e6430e332545512e15\": container with ID starting with 0e000c223f00245195ed22c76bd1cccfe484003e672ed0e6430e332545512e15 not found: ID does not exist"
Sep 16 18:34:30.806696 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.806671    2577 scope.go:117] "RemoveContainer" containerID="a0f2d27fceaefe70986a9d25846daff06162812abb1f2868af82344007362d88"
Sep 16 18:34:30.806952 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.806928    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0f2d27fceaefe70986a9d25846daff06162812abb1f2868af82344007362d88"} err="failed to get container status \"a0f2d27fceaefe70986a9d25846daff06162812abb1f2868af82344007362d88\": rpc error: code = NotFound desc = could not find container \"a0f2d27fceaefe70986a9d25846daff06162812abb1f2868af82344007362d88\": container with ID starting with a0f2d27fceaefe70986a9d25846daff06162812abb1f2868af82344007362d88 not found: ID does not exist"
Sep 16 18:34:30.807054 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.806952    2577 scope.go:117] "RemoveContainer" containerID="ab23fffced3d673a125f562e4a2801d48ed598312b1b351a0f51c60f90fb69b9"
Sep 16 18:34:30.807195 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.807175    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab23fffced3d673a125f562e4a2801d48ed598312b1b351a0f51c60f90fb69b9"} err="failed to get container status \"ab23fffced3d673a125f562e4a2801d48ed598312b1b351a0f51c60f90fb69b9\": rpc error: code = NotFound desc = could not find container \"ab23fffced3d673a125f562e4a2801d48ed598312b1b351a0f51c60f90fb69b9\": container with ID starting with ab23fffced3d673a125f562e4a2801d48ed598312b1b351a0f51c60f90fb69b9 not found: ID does not exist"
Sep 16 18:34:30.807245 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.807197    2577 scope.go:117] "RemoveContainer" containerID="0e000c223f00245195ed22c76bd1cccfe484003e672ed0e6430e332545512e15"
Sep 16 18:34:30.807405 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.807385    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e000c223f00245195ed22c76bd1cccfe484003e672ed0e6430e332545512e15"} err="failed to get container status \"0e000c223f00245195ed22c76bd1cccfe484003e672ed0e6430e332545512e15\": rpc error: code = NotFound desc = could not find container \"0e000c223f00245195ed22c76bd1cccfe484003e672ed0e6430e332545512e15\": container with ID starting with 0e000c223f00245195ed22c76bd1cccfe484003e672ed0e6430e332545512e15 not found: ID does not exist"
Sep 16 18:34:30.811846 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.811825    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"]
Sep 16 18:34:30.818875 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:30.818845    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-79c9df86d6cxpvk"]
Sep 16 18:34:31.081701 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:31.081658    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a9aae77-6433-4601-bdaa-88abe763dee7" path="/var/lib/kubelet/pods/8a9aae77-6433-4601-bdaa-88abe763dee7/volumes"
Sep 16 18:34:35.468709 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:35.468672    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:35.569609 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:35.569575    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:36.076294 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.076255    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"]
Sep 16 18:34:36.076776 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.076758    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a9aae77-6433-4601-bdaa-88abe763dee7" containerName="storage-initializer"
Sep 16 18:34:36.076829 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.076779    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a9aae77-6433-4601-bdaa-88abe763dee7" containerName="storage-initializer"
Sep 16 18:34:36.076829 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.076788    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6faad72-a48b-4dbb-9b7f-67226a813f4f" containerName="storage-initializer"
Sep 16 18:34:36.076829 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.076793    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6faad72-a48b-4dbb-9b7f-67226a813f4f" containerName="storage-initializer"
Sep 16 18:34:36.076829 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.076803    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6faad72-a48b-4dbb-9b7f-67226a813f4f" containerName="main"
Sep 16 18:34:36.076829 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.076809    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6faad72-a48b-4dbb-9b7f-67226a813f4f" containerName="main"
Sep 16 18:34:36.076829 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.076823    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a9aae77-6433-4601-bdaa-88abe763dee7" containerName="main"
Sep 16 18:34:36.076829 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.076829    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a9aae77-6433-4601-bdaa-88abe763dee7" containerName="main"
Sep 16 18:34:36.077083 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.076840    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a9aae77-6433-4601-bdaa-88abe763dee7" containerName="llm-d-routing-sidecar"
Sep 16 18:34:36.077083 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.076845    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a9aae77-6433-4601-bdaa-88abe763dee7" containerName="llm-d-routing-sidecar"
Sep 16 18:34:36.077083 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.076926    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a9aae77-6433-4601-bdaa-88abe763dee7" containerName="llm-d-routing-sidecar"
Sep 16 18:34:36.077083 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.076933    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a9aae77-6433-4601-bdaa-88abe763dee7" containerName="main"
Sep 16 18:34:36.077083 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.076940    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6faad72-a48b-4dbb-9b7f-67226a813f4f" containerName="main"
Sep 16 18:34:36.082007 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.081983    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"
Sep 16 18:34:36.085501 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.085481    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Sep 16 18:34:36.085736 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.085712    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:34:36.085851 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.085758    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Sep 16 18:34:36.085851 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.085769    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Sep 16 18:34:36.085851 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.085825    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-ln4h4\""
Sep 16 18:34:36.086080 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.085769    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Sep 16 18:34:36.093701 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.093675    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"]
Sep 16 18:34:36.146593 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.146556    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:34:36.163632 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.163592    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dc5eaf8-482d-40e2-ba80-cf932d716b48-dshm\") pod \"lora-multiple-adapters-test-kserve-6d768649b9-wks26\" (UID: \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"
Sep 16 18:34:36.163810 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.163640    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2dc5eaf8-482d-40e2-ba80-cf932d716b48-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6d768649b9-wks26\" (UID: \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"
Sep 16 18:34:36.163810 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.163715    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dc5eaf8-482d-40e2-ba80-cf932d716b48-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6d768649b9-wks26\" (UID: \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"
Sep 16 18:34:36.163810 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.163763    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bhl2\" (UniqueName: \"kubernetes.io/projected/2dc5eaf8-482d-40e2-ba80-cf932d716b48-kube-api-access-9bhl2\") pod \"lora-multiple-adapters-test-kserve-6d768649b9-wks26\" (UID: \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"
Sep 16 18:34:36.163988 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.163813    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dc5eaf8-482d-40e2-ba80-cf932d716b48-model-cache\") pod \"lora-multiple-adapters-test-kserve-6d768649b9-wks26\" (UID: \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"
Sep 16 18:34:36.163988 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.163838    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dc5eaf8-482d-40e2-ba80-cf932d716b48-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6d768649b9-wks26\" (UID: \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"
Sep 16 18:34:36.163988 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.163931    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dc5eaf8-482d-40e2-ba80-cf932d716b48-home\") pod \"lora-multiple-adapters-test-kserve-6d768649b9-wks26\" (UID: \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"
Sep 16 18:34:36.163988 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.163954    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc5eaf8-482d-40e2-ba80-cf932d716b48-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6d768649b9-wks26\" (UID: \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"
Sep 16 18:34:36.264758 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.264718    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dc5eaf8-482d-40e2-ba80-cf932d716b48-dshm\") pod \"lora-multiple-adapters-test-kserve-6d768649b9-wks26\" (UID: \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"
Sep 16 18:34:36.264758 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.264762    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2dc5eaf8-482d-40e2-ba80-cf932d716b48-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6d768649b9-wks26\" (UID: \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"
Sep 16 18:34:36.265062 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.264807    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dc5eaf8-482d-40e2-ba80-cf932d716b48-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6d768649b9-wks26\" (UID: \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"
Sep 16 18:34:36.265062 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.264903    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bhl2\" (UniqueName: \"kubernetes.io/projected/2dc5eaf8-482d-40e2-ba80-cf932d716b48-kube-api-access-9bhl2\") pod \"lora-multiple-adapters-test-kserve-6d768649b9-wks26\" (UID: \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"
Sep 16 18:34:36.265062 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.264950    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dc5eaf8-482d-40e2-ba80-cf932d716b48-model-cache\") pod \"lora-multiple-adapters-test-kserve-6d768649b9-wks26\" (UID: \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"
Sep 16 18:34:36.265062 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.264978    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dc5eaf8-482d-40e2-ba80-cf932d716b48-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6d768649b9-wks26\" (UID: \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"
Sep 16 18:34:36.265062 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.265031    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dc5eaf8-482d-40e2-ba80-cf932d716b48-home\") pod \"lora-multiple-adapters-test-kserve-6d768649b9-wks26\" (UID: \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"
Sep 16 18:34:36.265062 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.265055    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc5eaf8-482d-40e2-ba80-cf932d716b48-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6d768649b9-wks26\" (UID: \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"
Sep 16 18:34:36.265394 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.265256    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dc5eaf8-482d-40e2-ba80-cf932d716b48-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6d768649b9-wks26\" (UID: \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"
Sep 16 18:34:36.265394 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.265336    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dc5eaf8-482d-40e2-ba80-cf932d716b48-model-cache\") pod \"lora-multiple-adapters-test-kserve-6d768649b9-wks26\" (UID: \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"
Sep 16 18:34:36.265511 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.265479    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dc5eaf8-482d-40e2-ba80-cf932d716b48-home\") pod \"lora-multiple-adapters-test-kserve-6d768649b9-wks26\" (UID: \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"
Sep 16 18:34:36.265574 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.265489    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dc5eaf8-482d-40e2-ba80-cf932d716b48-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6d768649b9-wks26\" (UID: \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"
Sep 16 18:34:36.265693 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.265645    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2dc5eaf8-482d-40e2-ba80-cf932d716b48-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6d768649b9-wks26\" (UID: \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"
Sep 16 18:34:36.267083 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.267060    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dc5eaf8-482d-40e2-ba80-cf932d716b48-dshm\") pod \"lora-multiple-adapters-test-kserve-6d768649b9-wks26\" (UID: \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"
Sep 16 18:34:36.267503 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.267487    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc5eaf8-482d-40e2-ba80-cf932d716b48-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6d768649b9-wks26\" (UID: \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"
Sep 16 18:34:36.273569 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.273545    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bhl2\" (UniqueName: \"kubernetes.io/projected/2dc5eaf8-482d-40e2-ba80-cf932d716b48-kube-api-access-9bhl2\") pod \"lora-multiple-adapters-test-kserve-6d768649b9-wks26\" (UID: \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"
Sep 16 18:34:36.393884 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.393768    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"
Sep 16 18:34:36.690724 ip-10-0-136-122 kubenswrapper[2577]: W0916 18:34:36.690683    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc5eaf8_482d_40e2_ba80_cf932d716b48.slice/crio-8fd64f571cf26feb08ce9b5601666ea18e5783f4972ca4b689cf45626e859d45 WatchSource:0}: Error finding container 8fd64f571cf26feb08ce9b5601666ea18e5783f4972ca4b689cf45626e859d45: Status 404 returned error can't find the container with id 8fd64f571cf26feb08ce9b5601666ea18e5783f4972ca4b689cf45626e859d45
Sep 16 18:34:36.692937 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.692907    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"]
Sep 16 18:34:36.813113 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.813063    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26" event={"ID":"2dc5eaf8-482d-40e2-ba80-cf932d716b48","Type":"ContainerStarted","Data":"d492a2bae08c6f9c714edcecc9e4c09bf854bcdf911ef529d2a8fc3480a6008c"}
Sep 16 18:34:36.813318 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:36.813125    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26" event={"ID":"2dc5eaf8-482d-40e2-ba80-cf932d716b48","Type":"ContainerStarted","Data":"8fd64f571cf26feb08ce9b5601666ea18e5783f4972ca4b689cf45626e859d45"}
Sep 16 18:34:39.831354 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:39.831320    2577 generic.go:356] "Generic (PLEG): container finished" podID="2dc5eaf8-482d-40e2-ba80-cf932d716b48" containerID="d492a2bae08c6f9c714edcecc9e4c09bf854bcdf911ef529d2a8fc3480a6008c" exitCode=0
Sep 16 18:34:39.831671 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:39.831389    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26" event={"ID":"2dc5eaf8-482d-40e2-ba80-cf932d716b48","Type":"ContainerDied","Data":"d492a2bae08c6f9c714edcecc9e4c09bf854bcdf911ef529d2a8fc3480a6008c"}
Sep 16 18:34:40.838792 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:40.838756    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26" event={"ID":"2dc5eaf8-482d-40e2-ba80-cf932d716b48","Type":"ContainerStarted","Data":"091691a5c96fd0e4d02c06703a39c5c05daa773abb3508f74d03734cdc70c518"}
Sep 16 18:34:40.865676 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:40.865611    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26" podStartSLOduration=1.853519771 podStartE2EDuration="4.865592273s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.012072502s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:34:40.864327698 +0000 UTC m=+4202.227006504" watchObservedRunningTime="2026-09-16 18:34:40.865592273 +0000 UTC m=+4202.228271061"
Sep 16 18:34:41.394426 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:41.394382    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"
Sep 16 18:34:46.394303 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:46.394270    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"
Sep 16 18:34:46.396054 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:46.396019    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26" podUID="2dc5eaf8-482d-40e2-ba80-cf932d716b48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused"
Sep 16 18:34:56.394343 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:34:56.394305    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26" podUID="2dc5eaf8-482d-40e2-ba80-cf932d716b48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused"
Sep 16 18:35:04.755361 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:04.755274    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:06.152182 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:06.152145    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:06.394480 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:06.394441    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26" podUID="2dc5eaf8-482d-40e2-ba80-cf932d716b48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused"
Sep 16 18:35:08.066460 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.066426    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"]
Sep 16 18:35:08.070702 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.070677    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"
Sep 16 18:35:08.073806 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.073774    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 16 18:35:08.073961 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.073780    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 16 18:35:08.074153 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.074133    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 16 18:35:08.074258 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.074162    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:35:08.074258 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.074226    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-dnj2m\""
Sep 16 18:35:08.074375 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.074271    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 16 18:35:08.086230 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.086208    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"]
Sep 16 18:35:08.120466 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.120437    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:08.155835 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.155796    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3e4f08e-2944-42dd-ac12-50c42e93113a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq\" (UID: \"c3e4f08e-2944-42dd-ac12-50c42e93113a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"
Sep 16 18:35:08.156006 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.155847    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3e4f08e-2944-42dd-ac12-50c42e93113a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq\" (UID: \"c3e4f08e-2944-42dd-ac12-50c42e93113a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"
Sep 16 18:35:08.156006 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.155882    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5gbd\" (UniqueName: \"kubernetes.io/projected/c3e4f08e-2944-42dd-ac12-50c42e93113a-kube-api-access-j5gbd\") pod \"custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq\" (UID: \"c3e4f08e-2944-42dd-ac12-50c42e93113a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"
Sep 16 18:35:08.156006 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.155933    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3e4f08e-2944-42dd-ac12-50c42e93113a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq\" (UID: \"c3e4f08e-2944-42dd-ac12-50c42e93113a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"
Sep 16 18:35:08.156006 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.155951    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3e4f08e-2944-42dd-ac12-50c42e93113a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq\" (UID: \"c3e4f08e-2944-42dd-ac12-50c42e93113a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"
Sep 16 18:35:08.156006 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.155973    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3e4f08e-2944-42dd-ac12-50c42e93113a-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq\" (UID: \"c3e4f08e-2944-42dd-ac12-50c42e93113a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"
Sep 16 18:35:08.156006 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.156001    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3e4f08e-2944-42dd-ac12-50c42e93113a-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq\" (UID: \"c3e4f08e-2944-42dd-ac12-50c42e93113a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"
Sep 16 18:35:08.156257 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.156022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3e4f08e-2944-42dd-ac12-50c42e93113a-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq\" (UID: \"c3e4f08e-2944-42dd-ac12-50c42e93113a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"
Sep 16 18:35:08.257098 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.257061    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3e4f08e-2944-42dd-ac12-50c42e93113a-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq\" (UID: \"c3e4f08e-2944-42dd-ac12-50c42e93113a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"
Sep 16 18:35:08.257278 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.257106    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3e4f08e-2944-42dd-ac12-50c42e93113a-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq\" (UID: \"c3e4f08e-2944-42dd-ac12-50c42e93113a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"
Sep 16 18:35:08.257278 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.257155    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3e4f08e-2944-42dd-ac12-50c42e93113a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq\" (UID: \"c3e4f08e-2944-42dd-ac12-50c42e93113a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"
Sep 16 18:35:08.257278 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.257187    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3e4f08e-2944-42dd-ac12-50c42e93113a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq\" (UID: \"c3e4f08e-2944-42dd-ac12-50c42e93113a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"
Sep 16 18:35:08.257409 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.257343    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5gbd\" (UniqueName: \"kubernetes.io/projected/c3e4f08e-2944-42dd-ac12-50c42e93113a-kube-api-access-j5gbd\") pod \"custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq\" (UID: \"c3e4f08e-2944-42dd-ac12-50c42e93113a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"
Sep 16 18:35:08.257476 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.257417    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3e4f08e-2944-42dd-ac12-50c42e93113a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq\" (UID: \"c3e4f08e-2944-42dd-ac12-50c42e93113a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"
Sep 16 18:35:08.257476 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.257449    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3e4f08e-2944-42dd-ac12-50c42e93113a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq\" (UID: \"c3e4f08e-2944-42dd-ac12-50c42e93113a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"
Sep 16 18:35:08.257587 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.257481    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3e4f08e-2944-42dd-ac12-50c42e93113a-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq\" (UID: \"c3e4f08e-2944-42dd-ac12-50c42e93113a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"
Sep 16 18:35:08.257641 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.257603    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3e4f08e-2944-42dd-ac12-50c42e93113a-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq\" (UID: \"c3e4f08e-2944-42dd-ac12-50c42e93113a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"
Sep 16 18:35:08.257690 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.257652    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3e4f08e-2944-42dd-ac12-50c42e93113a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq\" (UID: \"c3e4f08e-2944-42dd-ac12-50c42e93113a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"
Sep 16 18:35:08.257690 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.257677    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3e4f08e-2944-42dd-ac12-50c42e93113a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq\" (UID: \"c3e4f08e-2944-42dd-ac12-50c42e93113a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"
Sep 16 18:35:08.257967 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.257938    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3e4f08e-2944-42dd-ac12-50c42e93113a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq\" (UID: \"c3e4f08e-2944-42dd-ac12-50c42e93113a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"
Sep 16 18:35:08.257967 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.257960    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3e4f08e-2944-42dd-ac12-50c42e93113a-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq\" (UID: \"c3e4f08e-2944-42dd-ac12-50c42e93113a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"
Sep 16 18:35:08.259887 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.259854    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3e4f08e-2944-42dd-ac12-50c42e93113a-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq\" (UID: \"c3e4f08e-2944-42dd-ac12-50c42e93113a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"
Sep 16 18:35:08.260304 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.260283    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3e4f08e-2944-42dd-ac12-50c42e93113a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq\" (UID: \"c3e4f08e-2944-42dd-ac12-50c42e93113a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"
Sep 16 18:35:08.267140 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.267116    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5gbd\" (UniqueName: \"kubernetes.io/projected/c3e4f08e-2944-42dd-ac12-50c42e93113a-kube-api-access-j5gbd\") pod \"custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq\" (UID: \"c3e4f08e-2944-42dd-ac12-50c42e93113a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"
Sep 16 18:35:08.382018 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.381919    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"
Sep 16 18:35:08.409362 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.409317    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:08.669392 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.669356    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"]
Sep 16 18:35:08.672425 ip-10-0-136-122 kubenswrapper[2577]: W0916 18:35:08.672397    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e4f08e_2944_42dd_ac12_50c42e93113a.slice/crio-0a8f851573596adf1643eceeac285decc4d31dd759e2b11eec250b9d25d74d30 WatchSource:0}: Error finding container 0a8f851573596adf1643eceeac285decc4d31dd759e2b11eec250b9d25d74d30: Status 404 returned error can't find the container with id 0a8f851573596adf1643eceeac285decc4d31dd759e2b11eec250b9d25d74d30
Sep 16 18:35:08.997933 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.997825    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq" event={"ID":"c3e4f08e-2944-42dd-ac12-50c42e93113a","Type":"ContainerStarted","Data":"171a10b8beac22268ae4d1591365f8a8f2872b280cfc16790b23131d51592afc"}
Sep 16 18:35:08.998082 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:08.997929    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq" event={"ID":"c3e4f08e-2944-42dd-ac12-50c42e93113a","Type":"ContainerStarted","Data":"0a8f851573596adf1643eceeac285decc4d31dd759e2b11eec250b9d25d74d30"}
Sep 16 18:35:12.020651 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:12.020613    2577 generic.go:356] "Generic (PLEG): container finished" podID="c3e4f08e-2944-42dd-ac12-50c42e93113a" containerID="171a10b8beac22268ae4d1591365f8a8f2872b280cfc16790b23131d51592afc" exitCode=0
Sep 16 18:35:12.021057 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:12.020675    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq" event={"ID":"c3e4f08e-2944-42dd-ac12-50c42e93113a","Type":"ContainerDied","Data":"171a10b8beac22268ae4d1591365f8a8f2872b280cfc16790b23131d51592afc"}
Sep 16 18:35:12.351800 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:12.351762    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:13.029622 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:13.029579    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq" event={"ID":"c3e4f08e-2944-42dd-ac12-50c42e93113a","Type":"ContainerStarted","Data":"a567768b078e891dbe6388ce3702aa5025b60449e1e81cf682a5f9a85debc3c6"}
Sep 16 18:35:13.062824 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:13.062766    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq" podStartSLOduration=2.622962418 podStartE2EDuration="5.062751138s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.43978872s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:35:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:35:13.061651121 +0000 UTC m=+4234.424329911" watchObservedRunningTime="2026-09-16 18:35:13.062751138 +0000 UTC m=+4234.425429924"
Sep 16 18:35:16.395110 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:16.395052    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26" podUID="2dc5eaf8-482d-40e2-ba80-cf932d716b48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused"
Sep 16 18:35:18.244230 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:18.244187    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:18.382533 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:18.382484    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"
Sep 16 18:35:18.382741 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:18.382576    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"
Sep 16 18:35:18.384433 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:18.384400    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq" podUID="c3e4f08e-2944-42dd-ac12-50c42e93113a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": dial tcp 10.133.0.91:8000: connect: connection refused"
Sep 16 18:35:18.788923 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:18.788886    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:18.860054 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:18.859988    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:22.770425 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:22.770378    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:26.394598 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:26.394553    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26" podUID="2dc5eaf8-482d-40e2-ba80-cf932d716b48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused"
Sep 16 18:35:28.383431 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:28.383387    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq" podUID="c3e4f08e-2944-42dd-ac12-50c42e93113a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": dial tcp 10.133.0.91:8000: connect: connection refused"
Sep 16 18:35:36.394848 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:36.394802    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26" podUID="2dc5eaf8-482d-40e2-ba80-cf932d716b48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused"
Sep 16 18:35:38.382849 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:38.382806    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq" podUID="c3e4f08e-2944-42dd-ac12-50c42e93113a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": dial tcp 10.133.0.91:8000: connect: connection refused"
Sep 16 18:35:43.516190 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:43.516154    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:46.395261 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:46.395223    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26" podUID="2dc5eaf8-482d-40e2-ba80-cf932d716b48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused"
Sep 16 18:35:48.383433 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:48.383392    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq" podUID="c3e4f08e-2944-42dd-ac12-50c42e93113a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": dial tcp 10.133.0.91:8000: connect: connection refused"
Sep 16 18:35:55.908740 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:55.908705    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:35:56.404490 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:56.404455    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"
Sep 16 18:35:56.412275 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:56.412241    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"
Sep 16 18:35:58.383299 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:35:58.383248    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq" podUID="c3e4f08e-2944-42dd-ac12-50c42e93113a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": dial tcp 10.133.0.91:8000: connect: connection refused"
Sep 16 18:36:08.383129 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:08.383063    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq" podUID="c3e4f08e-2944-42dd-ac12-50c42e93113a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": dial tcp 10.133.0.91:8000: connect: connection refused"
Sep 16 18:36:18.383201 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:18.383150    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq" podUID="c3e4f08e-2944-42dd-ac12-50c42e93113a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": dial tcp 10.133.0.91:8000: connect: connection refused"
Sep 16 18:36:25.460179 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:25.460142    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-6d768649b9-wks26_2dc5eaf8-482d-40e2-ba80-cf932d716b48/storage-initializer/0.log"
Sep 16 18:36:25.473949 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:25.473900    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-6d768649b9-wks26_2dc5eaf8-482d-40e2-ba80-cf932d716b48/main/0.log"
Sep 16 18:36:26.769187 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:26.769143    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"]
Sep 16 18:36:26.788087 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:26.788055    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:28.383170 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:28.383130    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq" podUID="c3e4f08e-2944-42dd-ac12-50c42e93113a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": dial tcp 10.133.0.91:8000: connect: connection refused"
Sep 16 18:36:38.382492 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:38.382394    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq" podUID="c3e4f08e-2944-42dd-ac12-50c42e93113a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": dial tcp 10.133.0.91:8000: connect: connection refused"
Sep 16 18:36:41.792303 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:41.792236    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26" podUID="2dc5eaf8-482d-40e2-ba80-cf932d716b48" containerName="main" containerID="cri-o://091691a5c96fd0e4d02c06703a39c5c05daa773abb3508f74d03734cdc70c518" gracePeriod=45
Sep 16 18:36:42.405876 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:42.405817    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26" podUID="2dc5eaf8-482d-40e2-ba80-cf932d716b48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": dial tcp 10.133.0.90:8000: connect: connection refused"
Sep 16 18:36:42.756851 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:42.756819    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"
Sep 16 18:36:42.822194 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:42.822163    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dc5eaf8-482d-40e2-ba80-cf932d716b48-dshm\") pod \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\" (UID: \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\") "
Sep 16 18:36:42.822676 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:42.822207    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2dc5eaf8-482d-40e2-ba80-cf932d716b48-cabundle-cert\") pod \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\" (UID: \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\") "
Sep 16 18:36:42.822676 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:42.822227    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bhl2\" (UniqueName: \"kubernetes.io/projected/2dc5eaf8-482d-40e2-ba80-cf932d716b48-kube-api-access-9bhl2\") pod \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\" (UID: \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\") "
Sep 16 18:36:42.822676 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:42.822244    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dc5eaf8-482d-40e2-ba80-cf932d716b48-home\") pod \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\" (UID: \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\") "
Sep 16 18:36:42.822676 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:42.822272    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dc5eaf8-482d-40e2-ba80-cf932d716b48-tmp-dir\") pod \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\" (UID: \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\") "
Sep 16 18:36:42.822676 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:42.822397    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc5eaf8-482d-40e2-ba80-cf932d716b48-tls-certs\") pod \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\" (UID: \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\") "
Sep 16 18:36:42.822676 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:42.822510    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dc5eaf8-482d-40e2-ba80-cf932d716b48-model-cache\") pod \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\" (UID: \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\") "
Sep 16 18:36:42.822676 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:42.822550    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dc5eaf8-482d-40e2-ba80-cf932d716b48-kserve-provision-location\") pod \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\" (UID: \"2dc5eaf8-482d-40e2-ba80-cf932d716b48\") "
Sep 16 18:36:42.822676 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:42.822640    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dc5eaf8-482d-40e2-ba80-cf932d716b48-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2dc5eaf8-482d-40e2-ba80-cf932d716b48" (UID: "2dc5eaf8-482d-40e2-ba80-cf932d716b48"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:36:42.822676 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:42.822670    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc5eaf8-482d-40e2-ba80-cf932d716b48-home" (OuterVolumeSpecName: "home") pod "2dc5eaf8-482d-40e2-ba80-cf932d716b48" (UID: "2dc5eaf8-482d-40e2-ba80-cf932d716b48"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:36:42.823193 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:42.822969    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2dc5eaf8-482d-40e2-ba80-cf932d716b48-cabundle-cert\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:36:42.823193 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:42.822992    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dc5eaf8-482d-40e2-ba80-cf932d716b48-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:36:42.823193 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:42.822990    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc5eaf8-482d-40e2-ba80-cf932d716b48-model-cache" (OuterVolumeSpecName: "model-cache") pod "2dc5eaf8-482d-40e2-ba80-cf932d716b48" (UID: "2dc5eaf8-482d-40e2-ba80-cf932d716b48"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:36:42.825262 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:42.825232    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dc5eaf8-482d-40e2-ba80-cf932d716b48-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2dc5eaf8-482d-40e2-ba80-cf932d716b48" (UID: "2dc5eaf8-482d-40e2-ba80-cf932d716b48"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:36:42.825262 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:42.825243    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc5eaf8-482d-40e2-ba80-cf932d716b48-kube-api-access-9bhl2" (OuterVolumeSpecName: "kube-api-access-9bhl2") pod "2dc5eaf8-482d-40e2-ba80-cf932d716b48" (UID: "2dc5eaf8-482d-40e2-ba80-cf932d716b48"). InnerVolumeSpecName "kube-api-access-9bhl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:36:42.825483 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:42.825461    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc5eaf8-482d-40e2-ba80-cf932d716b48-dshm" (OuterVolumeSpecName: "dshm") pod "2dc5eaf8-482d-40e2-ba80-cf932d716b48" (UID: "2dc5eaf8-482d-40e2-ba80-cf932d716b48"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:36:42.834465 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:42.834438    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc5eaf8-482d-40e2-ba80-cf932d716b48-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2dc5eaf8-482d-40e2-ba80-cf932d716b48" (UID: "2dc5eaf8-482d-40e2-ba80-cf932d716b48"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:36:42.848611 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:42.848580    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc5eaf8-482d-40e2-ba80-cf932d716b48-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2dc5eaf8-482d-40e2-ba80-cf932d716b48" (UID: "2dc5eaf8-482d-40e2-ba80-cf932d716b48"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:36:42.924243 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:42.924151    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dc5eaf8-482d-40e2-ba80-cf932d716b48-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:36:42.924243 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:42.924184    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bhl2\" (UniqueName: \"kubernetes.io/projected/2dc5eaf8-482d-40e2-ba80-cf932d716b48-kube-api-access-9bhl2\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:36:42.924243 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:42.924206    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dc5eaf8-482d-40e2-ba80-cf932d716b48-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:36:42.924243 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:42.924216    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc5eaf8-482d-40e2-ba80-cf932d716b48-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:36:42.924243 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:42.924224    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dc5eaf8-482d-40e2-ba80-cf932d716b48-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:36:42.924243 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:42.924232    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dc5eaf8-482d-40e2-ba80-cf932d716b48-kserve-provision-location\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:36:43.566413 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:43.566375    2577 generic.go:356] "Generic (PLEG): container finished" podID="2dc5eaf8-482d-40e2-ba80-cf932d716b48" containerID="091691a5c96fd0e4d02c06703a39c5c05daa773abb3508f74d03734cdc70c518" exitCode=0
Sep 16 18:36:43.566617 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:43.566411    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26" event={"ID":"2dc5eaf8-482d-40e2-ba80-cf932d716b48","Type":"ContainerDied","Data":"091691a5c96fd0e4d02c06703a39c5c05daa773abb3508f74d03734cdc70c518"}
Sep 16 18:36:43.566617 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:43.566434    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"
Sep 16 18:36:43.566617 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:43.566480    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26" event={"ID":"2dc5eaf8-482d-40e2-ba80-cf932d716b48","Type":"ContainerDied","Data":"8fd64f571cf26feb08ce9b5601666ea18e5783f4972ca4b689cf45626e859d45"}
Sep 16 18:36:43.566617 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:43.566496    2577 scope.go:117] "RemoveContainer" containerID="091691a5c96fd0e4d02c06703a39c5c05daa773abb3508f74d03734cdc70c518"
Sep 16 18:36:43.577348 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:43.577328    2577 scope.go:117] "RemoveContainer" containerID="d492a2bae08c6f9c714edcecc9e4c09bf854bcdf911ef529d2a8fc3480a6008c"
Sep 16 18:36:43.585996 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:43.585835    2577 scope.go:117] "RemoveContainer" containerID="091691a5c96fd0e4d02c06703a39c5c05daa773abb3508f74d03734cdc70c518"
Sep 16 18:36:43.586221 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:36:43.586174    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"091691a5c96fd0e4d02c06703a39c5c05daa773abb3508f74d03734cdc70c518\": container with ID starting with 091691a5c96fd0e4d02c06703a39c5c05daa773abb3508f74d03734cdc70c518 not found: ID does not exist" containerID="091691a5c96fd0e4d02c06703a39c5c05daa773abb3508f74d03734cdc70c518"
Sep 16 18:36:43.586306 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:43.586217    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"091691a5c96fd0e4d02c06703a39c5c05daa773abb3508f74d03734cdc70c518"} err="failed to get container status \"091691a5c96fd0e4d02c06703a39c5c05daa773abb3508f74d03734cdc70c518\": rpc error: code = NotFound desc = could not find container \"091691a5c96fd0e4d02c06703a39c5c05daa773abb3508f74d03734cdc70c518\": container with ID starting with 091691a5c96fd0e4d02c06703a39c5c05daa773abb3508f74d03734cdc70c518 not found: ID does not exist"
Sep 16 18:36:43.586306 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:43.586243    2577 scope.go:117] "RemoveContainer" containerID="d492a2bae08c6f9c714edcecc9e4c09bf854bcdf911ef529d2a8fc3480a6008c"
Sep 16 18:36:43.586569 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:36:43.586537    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d492a2bae08c6f9c714edcecc9e4c09bf854bcdf911ef529d2a8fc3480a6008c\": container with ID starting with d492a2bae08c6f9c714edcecc9e4c09bf854bcdf911ef529d2a8fc3480a6008c not found: ID does not exist" containerID="d492a2bae08c6f9c714edcecc9e4c09bf854bcdf911ef529d2a8fc3480a6008c"
Sep 16 18:36:43.586697 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:43.586571    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d492a2bae08c6f9c714edcecc9e4c09bf854bcdf911ef529d2a8fc3480a6008c"} err="failed to get container status \"d492a2bae08c6f9c714edcecc9e4c09bf854bcdf911ef529d2a8fc3480a6008c\": rpc error: code = NotFound desc = could not find container \"d492a2bae08c6f9c714edcecc9e4c09bf854bcdf911ef529d2a8fc3480a6008c\": container with ID starting with d492a2bae08c6f9c714edcecc9e4c09bf854bcdf911ef529d2a8fc3480a6008c not found: ID does not exist"
Sep 16 18:36:43.588182 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:43.588157    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"]
Sep 16 18:36:43.594113 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:43.594090    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d768649b9-wks26"]
Sep 16 18:36:45.080330 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:45.080289    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dc5eaf8-482d-40e2-ba80-cf932d716b48" path="/var/lib/kubelet/pods/2dc5eaf8-482d-40e2-ba80-cf932d716b48/volumes"
Sep 16 18:36:46.969594 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:46.969558    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:47.271389 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:47.271297    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:48.383100 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:48.383058    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq" podUID="c3e4f08e-2944-42dd-ac12-50c42e93113a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": dial tcp 10.133.0.91:8000: connect: connection refused"
Sep 16 18:36:53.861254 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:53.861221    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:53.999245 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:53.999193    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:58.382716 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.382658    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq" podUID="c3e4f08e-2944-42dd-ac12-50c42e93113a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": dial tcp 10.133.0.91:8000: connect: connection refused"
Sep 16 18:36:58.779266 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.779161    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"]
Sep 16 18:36:58.779558 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.779542    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dc5eaf8-482d-40e2-ba80-cf932d716b48" containerName="storage-initializer"
Sep 16 18:36:58.779558 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.779557    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dc5eaf8-482d-40e2-ba80-cf932d716b48" containerName="storage-initializer"
Sep 16 18:36:58.779717 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.779570    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dc5eaf8-482d-40e2-ba80-cf932d716b48" containerName="main"
Sep 16 18:36:58.779717 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.779576    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dc5eaf8-482d-40e2-ba80-cf932d716b48" containerName="main"
Sep 16 18:36:58.779717 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.779652    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dc5eaf8-482d-40e2-ba80-cf932d716b48" containerName="main"
Sep 16 18:36:58.786409 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.786380    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"
Sep 16 18:36:58.789592 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.789564    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 16 18:36:58.789592 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.789583    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:36:58.789793 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.789591    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Sep 16 18:36:58.790046 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.790025    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 16 18:36:58.790046 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.790040    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-ztzl2\""
Sep 16 18:36:58.791016 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.790994    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 16 18:36:58.797039 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.797015    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"]
Sep 16 18:36:58.818910 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.818858    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:58.867095 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.867058    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-home\") pod \"tls-verification-test-kserve-55d67d44c5-jv87q\" (UID: \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"
Sep 16 18:36:58.867269 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.867145    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-tls-certs\") pod \"tls-verification-test-kserve-55d67d44c5-jv87q\" (UID: \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"
Sep 16 18:36:58.867269 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.867166    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9kmx\" (UniqueName: \"kubernetes.io/projected/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-kube-api-access-r9kmx\") pod \"tls-verification-test-kserve-55d67d44c5-jv87q\" (UID: \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"
Sep 16 18:36:58.867269 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.867192    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-dshm\") pod \"tls-verification-test-kserve-55d67d44c5-jv87q\" (UID: \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"
Sep 16 18:36:58.867269 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.867253    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-model-cache\") pod \"tls-verification-test-kserve-55d67d44c5-jv87q\" (UID: \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"
Sep 16 18:36:58.867430 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.867292    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-kserve-provision-location\") pod \"tls-verification-test-kserve-55d67d44c5-jv87q\" (UID: \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"
Sep 16 18:36:58.867430 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.867369    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-tmp-dir\") pod \"tls-verification-test-kserve-55d67d44c5-jv87q\" (UID: \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"
Sep 16 18:36:58.867430 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.867396    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-cabundle-cert\") pod \"tls-verification-test-kserve-55d67d44c5-jv87q\" (UID: \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"
Sep 16 18:36:58.967854 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.967814    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-tls-certs\") pod \"tls-verification-test-kserve-55d67d44c5-jv87q\" (UID: \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"
Sep 16 18:36:58.968038 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.967855    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9kmx\" (UniqueName: \"kubernetes.io/projected/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-kube-api-access-r9kmx\") pod \"tls-verification-test-kserve-55d67d44c5-jv87q\" (UID: \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"
Sep 16 18:36:58.968038 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.967895    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-dshm\") pod \"tls-verification-test-kserve-55d67d44c5-jv87q\" (UID: \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"
Sep 16 18:36:58.968038 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.967919    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-model-cache\") pod \"tls-verification-test-kserve-55d67d44c5-jv87q\" (UID: \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"
Sep 16 18:36:58.968038 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.967938    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-kserve-provision-location\") pod \"tls-verification-test-kserve-55d67d44c5-jv87q\" (UID: \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"
Sep 16 18:36:58.968038 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.967975    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-tmp-dir\") pod \"tls-verification-test-kserve-55d67d44c5-jv87q\" (UID: \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"
Sep 16 18:36:58.968038 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.968004    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-cabundle-cert\") pod \"tls-verification-test-kserve-55d67d44c5-jv87q\" (UID: \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"
Sep 16 18:36:58.968361 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.968061    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-home\") pod \"tls-verification-test-kserve-55d67d44c5-jv87q\" (UID: \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"
Sep 16 18:36:58.968420 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.968360    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-model-cache\") pod \"tls-verification-test-kserve-55d67d44c5-jv87q\" (UID: \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"
Sep 16 18:36:58.968585 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.968482    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-tmp-dir\") pod \"tls-verification-test-kserve-55d67d44c5-jv87q\" (UID: \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"
Sep 16 18:36:58.968585 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.968577    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-kserve-provision-location\") pod \"tls-verification-test-kserve-55d67d44c5-jv87q\" (UID: \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"
Sep 16 18:36:58.968752 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.968683    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-home\") pod \"tls-verification-test-kserve-55d67d44c5-jv87q\" (UID: \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"
Sep 16 18:36:58.968876 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.968836    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-cabundle-cert\") pod \"tls-verification-test-kserve-55d67d44c5-jv87q\" (UID: \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"
Sep 16 18:36:58.970453 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.970427    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-dshm\") pod \"tls-verification-test-kserve-55d67d44c5-jv87q\" (UID: \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"
Sep 16 18:36:58.970605 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.970585    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-tls-certs\") pod \"tls-verification-test-kserve-55d67d44c5-jv87q\" (UID: \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"
Sep 16 18:36:58.979169 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:58.979146    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9kmx\" (UniqueName: \"kubernetes.io/projected/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-kube-api-access-r9kmx\") pod \"tls-verification-test-kserve-55d67d44c5-jv87q\" (UID: \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"
Sep 16 18:36:59.036919 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:59.036820    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:36:59.099197 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:59.099164    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"
Sep 16 18:36:59.425999 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:59.425783    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"]
Sep 16 18:36:59.429494 ip-10-0-136-122 kubenswrapper[2577]: W0916 18:36:59.429449    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a87dce8_bde4_42c5_aa2a_7a2f4c3c2f68.slice/crio-6200f59b6c3bdcccd62932990ecd85416407193ffc1cfcc88c2fc4cee896f8ca WatchSource:0}: Error finding container 6200f59b6c3bdcccd62932990ecd85416407193ffc1cfcc88c2fc4cee896f8ca: Status 404 returned error can't find the container with id 6200f59b6c3bdcccd62932990ecd85416407193ffc1cfcc88c2fc4cee896f8ca
Sep 16 18:36:59.665129 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:59.665088    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q" event={"ID":"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68","Type":"ContainerStarted","Data":"bb9b989bab98bc717147d69de42995dc220224f30a4668563797e161ab2461c3"}
Sep 16 18:36:59.665346 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:36:59.665155    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q" event={"ID":"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68","Type":"ContainerStarted","Data":"6200f59b6c3bdcccd62932990ecd85416407193ffc1cfcc88c2fc4cee896f8ca"}
Sep 16 18:37:02.682939 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:02.682902    2577 generic.go:356] "Generic (PLEG): container finished" podID="8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68" containerID="bb9b989bab98bc717147d69de42995dc220224f30a4668563797e161ab2461c3" exitCode=0
Sep 16 18:37:02.683341 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:02.682975    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q" event={"ID":"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68","Type":"ContainerDied","Data":"bb9b989bab98bc717147d69de42995dc220224f30a4668563797e161ab2461c3"}
Sep 16 18:37:03.691192 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:03.691151    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q" event={"ID":"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68","Type":"ContainerStarted","Data":"a31c2ea775ea74988170a58f30052a70f2c1f174b386e6e72aaaa43bbad5529c"}
Sep 16 18:37:03.726477 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:03.726411    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q" podStartSLOduration=3.177659824 podStartE2EDuration="5.726387838s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.548728014s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:36:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:37:03.722350102 +0000 UTC m=+4345.085028900" watchObservedRunningTime="2026-09-16 18:37:03.726387838 +0000 UTC m=+4345.089066632"
Sep 16 18:37:04.100021 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:04.099955    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"
Sep 16 18:37:08.393946 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:08.393909    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"
Sep 16 18:37:08.401978 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:08.401949    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"
Sep 16 18:37:08.842078 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:08.842036    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:09.100077 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:09.099977    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"
Sep 16 18:37:09.101910 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:09.101851    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q" podUID="8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 16 18:37:19.100036 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:19.099996    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q" podUID="8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 16 18:37:19.906815 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:19.906778    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"]
Sep 16 18:37:19.911190 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:19.911155    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:20.010739 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:20.010703    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:37:29.099577 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:29.099533    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q" podUID="8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 16 18:37:34.931006 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:34.930935    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq" podUID="c3e4f08e-2944-42dd-ac12-50c42e93113a" containerName="main" containerID="cri-o://a567768b078e891dbe6388ce3702aa5025b60449e1e81cf682a5f9a85debc3c6" gracePeriod=45
Sep 16 18:37:35.686440 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.686413    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"
Sep 16 18:37:35.792083 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.791983    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3e4f08e-2944-42dd-ac12-50c42e93113a-model-cache\") pod \"c3e4f08e-2944-42dd-ac12-50c42e93113a\" (UID: \"c3e4f08e-2944-42dd-ac12-50c42e93113a\") "
Sep 16 18:37:35.792083 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.792029    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3e4f08e-2944-42dd-ac12-50c42e93113a-home\") pod \"c3e4f08e-2944-42dd-ac12-50c42e93113a\" (UID: \"c3e4f08e-2944-42dd-ac12-50c42e93113a\") "
Sep 16 18:37:35.792083 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.792064    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3e4f08e-2944-42dd-ac12-50c42e93113a-kserve-provision-location\") pod \"c3e4f08e-2944-42dd-ac12-50c42e93113a\" (UID: \"c3e4f08e-2944-42dd-ac12-50c42e93113a\") "
Sep 16 18:37:35.792372 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.792102    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5gbd\" (UniqueName: \"kubernetes.io/projected/c3e4f08e-2944-42dd-ac12-50c42e93113a-kube-api-access-j5gbd\") pod \"c3e4f08e-2944-42dd-ac12-50c42e93113a\" (UID: \"c3e4f08e-2944-42dd-ac12-50c42e93113a\") "
Sep 16 18:37:35.792372 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.792127    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3e4f08e-2944-42dd-ac12-50c42e93113a-tmp-dir\") pod \"c3e4f08e-2944-42dd-ac12-50c42e93113a\" (UID: \"c3e4f08e-2944-42dd-ac12-50c42e93113a\") "
Sep 16 18:37:35.792372 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.792153    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3e4f08e-2944-42dd-ac12-50c42e93113a-cabundle-cert\") pod \"c3e4f08e-2944-42dd-ac12-50c42e93113a\" (UID: \"c3e4f08e-2944-42dd-ac12-50c42e93113a\") "
Sep 16 18:37:35.792372 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.792184    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3e4f08e-2944-42dd-ac12-50c42e93113a-tls-certs\") pod \"c3e4f08e-2944-42dd-ac12-50c42e93113a\" (UID: \"c3e4f08e-2944-42dd-ac12-50c42e93113a\") "
Sep 16 18:37:35.792372 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.792218    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3e4f08e-2944-42dd-ac12-50c42e93113a-dshm\") pod \"c3e4f08e-2944-42dd-ac12-50c42e93113a\" (UID: \"c3e4f08e-2944-42dd-ac12-50c42e93113a\") "
Sep 16 18:37:35.792372 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.792311    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e4f08e-2944-42dd-ac12-50c42e93113a-model-cache" (OuterVolumeSpecName: "model-cache") pod "c3e4f08e-2944-42dd-ac12-50c42e93113a" (UID: "c3e4f08e-2944-42dd-ac12-50c42e93113a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:37:35.793045 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.793014    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3e4f08e-2944-42dd-ac12-50c42e93113a-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:37:35.795466 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.793306    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e4f08e-2944-42dd-ac12-50c42e93113a-home" (OuterVolumeSpecName: "home") pod "c3e4f08e-2944-42dd-ac12-50c42e93113a" (UID: "c3e4f08e-2944-42dd-ac12-50c42e93113a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:37:35.795466 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.793746    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3e4f08e-2944-42dd-ac12-50c42e93113a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c3e4f08e-2944-42dd-ac12-50c42e93113a" (UID: "c3e4f08e-2944-42dd-ac12-50c42e93113a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:37:35.805678 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.804652    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e4f08e-2944-42dd-ac12-50c42e93113a-dshm" (OuterVolumeSpecName: "dshm") pod "c3e4f08e-2944-42dd-ac12-50c42e93113a" (UID: "c3e4f08e-2944-42dd-ac12-50c42e93113a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:37:35.805904 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.805726    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e4f08e-2944-42dd-ac12-50c42e93113a-kube-api-access-j5gbd" (OuterVolumeSpecName: "kube-api-access-j5gbd") pod "c3e4f08e-2944-42dd-ac12-50c42e93113a" (UID: "c3e4f08e-2944-42dd-ac12-50c42e93113a"). InnerVolumeSpecName "kube-api-access-j5gbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:37:35.807326 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.807299    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3e4f08e-2944-42dd-ac12-50c42e93113a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c3e4f08e-2944-42dd-ac12-50c42e93113a" (UID: "c3e4f08e-2944-42dd-ac12-50c42e93113a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:37:35.815141 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.815089    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e4f08e-2944-42dd-ac12-50c42e93113a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c3e4f08e-2944-42dd-ac12-50c42e93113a" (UID: "c3e4f08e-2944-42dd-ac12-50c42e93113a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:37:35.826182 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.826150    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e4f08e-2944-42dd-ac12-50c42e93113a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c3e4f08e-2944-42dd-ac12-50c42e93113a" (UID: "c3e4f08e-2944-42dd-ac12-50c42e93113a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:37:35.883261 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.883227    2577 generic.go:356] "Generic (PLEG): container finished" podID="c3e4f08e-2944-42dd-ac12-50c42e93113a" containerID="a567768b078e891dbe6388ce3702aa5025b60449e1e81cf682a5f9a85debc3c6" exitCode=0
Sep 16 18:37:35.883470 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.883293    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"
Sep 16 18:37:35.883470 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.883298    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq" event={"ID":"c3e4f08e-2944-42dd-ac12-50c42e93113a","Type":"ContainerDied","Data":"a567768b078e891dbe6388ce3702aa5025b60449e1e81cf682a5f9a85debc3c6"}
Sep 16 18:37:35.883470 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.883442    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq" event={"ID":"c3e4f08e-2944-42dd-ac12-50c42e93113a","Type":"ContainerDied","Data":"0a8f851573596adf1643eceeac285decc4d31dd759e2b11eec250b9d25d74d30"}
Sep 16 18:37:35.883470 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.883466    2577 scope.go:117] "RemoveContainer" containerID="a567768b078e891dbe6388ce3702aa5025b60449e1e81cf682a5f9a85debc3c6"
Sep 16 18:37:35.893504 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.893480    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5gbd\" (UniqueName: \"kubernetes.io/projected/c3e4f08e-2944-42dd-ac12-50c42e93113a-kube-api-access-j5gbd\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:37:35.893606 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.893522    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3e4f08e-2944-42dd-ac12-50c42e93113a-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:37:35.893606 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.893535    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3e4f08e-2944-42dd-ac12-50c42e93113a-cabundle-cert\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:37:35.893606 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.893543    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3e4f08e-2944-42dd-ac12-50c42e93113a-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:37:35.893606 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.893552    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3e4f08e-2944-42dd-ac12-50c42e93113a-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:37:35.893606 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.893559    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3e4f08e-2944-42dd-ac12-50c42e93113a-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:37:35.893606 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.893567    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3e4f08e-2944-42dd-ac12-50c42e93113a-kserve-provision-location\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:37:35.893947 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.893920    2577 scope.go:117] "RemoveContainer" containerID="171a10b8beac22268ae4d1591365f8a8f2872b280cfc16790b23131d51592afc"
Sep 16 18:37:35.906849 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.906826    2577 scope.go:117] "RemoveContainer" containerID="a567768b078e891dbe6388ce3702aa5025b60449e1e81cf682a5f9a85debc3c6"
Sep 16 18:37:35.907164 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:37:35.907138    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a567768b078e891dbe6388ce3702aa5025b60449e1e81cf682a5f9a85debc3c6\": container with ID starting with a567768b078e891dbe6388ce3702aa5025b60449e1e81cf682a5f9a85debc3c6 not found: ID does not exist" containerID="a567768b078e891dbe6388ce3702aa5025b60449e1e81cf682a5f9a85debc3c6"
Sep 16 18:37:35.907253 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.907176    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a567768b078e891dbe6388ce3702aa5025b60449e1e81cf682a5f9a85debc3c6"} err="failed to get container status \"a567768b078e891dbe6388ce3702aa5025b60449e1e81cf682a5f9a85debc3c6\": rpc error: code = NotFound desc = could not find container \"a567768b078e891dbe6388ce3702aa5025b60449e1e81cf682a5f9a85debc3c6\": container with ID starting with a567768b078e891dbe6388ce3702aa5025b60449e1e81cf682a5f9a85debc3c6 not found: ID does not exist"
Sep 16 18:37:35.907253 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.907200    2577 scope.go:117] "RemoveContainer" containerID="171a10b8beac22268ae4d1591365f8a8f2872b280cfc16790b23131d51592afc"
Sep 16 18:37:35.907467 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:37:35.907448    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"171a10b8beac22268ae4d1591365f8a8f2872b280cfc16790b23131d51592afc\": container with ID starting with 171a10b8beac22268ae4d1591365f8a8f2872b280cfc16790b23131d51592afc not found: ID does not exist" containerID="171a10b8beac22268ae4d1591365f8a8f2872b280cfc16790b23131d51592afc"
Sep 16 18:37:35.907524 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.907477    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"171a10b8beac22268ae4d1591365f8a8f2872b280cfc16790b23131d51592afc"} err="failed to get container status \"171a10b8beac22268ae4d1591365f8a8f2872b280cfc16790b23131d51592afc\": rpc error: code = NotFound desc = could not find container \"171a10b8beac22268ae4d1591365f8a8f2872b280cfc16790b23131d51592afc\": container with ID starting with 171a10b8beac22268ae4d1591365f8a8f2872b280cfc16790b23131d51592afc not found: ID does not exist"
Sep 16 18:37:35.917720 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.917683    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"]
Sep 16 18:37:35.921613 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:35.921590    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76c77b9f4c-b9cmq"]
Sep 16 18:37:37.081065 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:37.081033    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3e4f08e-2944-42dd-ac12-50c42e93113a" path="/var/lib/kubelet/pods/c3e4f08e-2944-42dd-ac12-50c42e93113a/volumes"
Sep 16 18:37:39.100491 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:39.100447    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q" podUID="8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 16 18:37:49.100392 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:49.100341    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q" podUID="8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 16 18:37:59.100016 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:37:59.099963    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q" podUID="8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 16 18:38:09.100454 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:09.100408    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q" podUID="8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 16 18:38:10.029194 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.029142    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:10.121495 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.121444    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:10.258152 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.258116    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"]
Sep 16 18:38:10.258671 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.258635    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3e4f08e-2944-42dd-ac12-50c42e93113a" containerName="main"
Sep 16 18:38:10.258671 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.258662    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3e4f08e-2944-42dd-ac12-50c42e93113a" containerName="main"
Sep 16 18:38:10.258897 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.258689    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3e4f08e-2944-42dd-ac12-50c42e93113a" containerName="storage-initializer"
Sep 16 18:38:10.258897 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.258698    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3e4f08e-2944-42dd-ac12-50c42e93113a" containerName="storage-initializer"
Sep 16 18:38:10.258897 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.258789    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3e4f08e-2944-42dd-ac12-50c42e93113a" containerName="main"
Sep 16 18:38:10.261918 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.261891    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:10.264837 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.264798    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-xxs5v\""
Sep 16 18:38:10.264970 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.264925    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 16 18:38:10.265062 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.265007    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Sep 16 18:38:10.265196 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.265179    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 16 18:38:10.276945 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.276920    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"]
Sep 16 18:38:10.292938 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.292845    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:10.388364 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.388332    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:10.392499 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.392467    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a85863-b59f-44f4-9d33-de426efa8221-workload-certs\") pod \"router-gateway-2-openshift-default-5c874f788-rjxkq\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:10.392499 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.392497    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a85863-b59f-44f4-9d33-de426efa8221-istio-token\") pod \"router-gateway-2-openshift-default-5c874f788-rjxkq\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:10.392708 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.392526    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a85863-b59f-44f4-9d33-de426efa8221-istio-podinfo\") pod \"router-gateway-2-openshift-default-5c874f788-rjxkq\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:10.392708 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.392556    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a85863-b59f-44f4-9d33-de426efa8221-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5c874f788-rjxkq\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:10.392708 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.392584    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a85863-b59f-44f4-9d33-de426efa8221-credential-socket\") pod \"router-gateway-2-openshift-default-5c874f788-rjxkq\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:10.392708 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.392607    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2m9g\" (UniqueName: \"kubernetes.io/projected/e0a85863-b59f-44f4-9d33-de426efa8221-kube-api-access-t2m9g\") pod \"router-gateway-2-openshift-default-5c874f788-rjxkq\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:10.392708 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.392633    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a85863-b59f-44f4-9d33-de426efa8221-workload-socket\") pod \"router-gateway-2-openshift-default-5c874f788-rjxkq\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:10.392708 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.392695    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a85863-b59f-44f4-9d33-de426efa8221-istio-envoy\") pod \"router-gateway-2-openshift-default-5c874f788-rjxkq\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:10.392991 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.392750    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a85863-b59f-44f4-9d33-de426efa8221-istio-data\") pod \"router-gateway-2-openshift-default-5c874f788-rjxkq\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:10.493312 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.493274    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a85863-b59f-44f4-9d33-de426efa8221-workload-certs\") pod \"router-gateway-2-openshift-default-5c874f788-rjxkq\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:10.493312 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.493312    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a85863-b59f-44f4-9d33-de426efa8221-istio-token\") pod \"router-gateway-2-openshift-default-5c874f788-rjxkq\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:10.493572 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.493351    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a85863-b59f-44f4-9d33-de426efa8221-istio-podinfo\") pod \"router-gateway-2-openshift-default-5c874f788-rjxkq\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:10.493572 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.493375    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a85863-b59f-44f4-9d33-de426efa8221-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5c874f788-rjxkq\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:10.493572 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.493414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a85863-b59f-44f4-9d33-de426efa8221-credential-socket\") pod \"router-gateway-2-openshift-default-5c874f788-rjxkq\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:10.493572 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.493453    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2m9g\" (UniqueName: \"kubernetes.io/projected/e0a85863-b59f-44f4-9d33-de426efa8221-kube-api-access-t2m9g\") pod \"router-gateway-2-openshift-default-5c874f788-rjxkq\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:10.493572 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.493479    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a85863-b59f-44f4-9d33-de426efa8221-workload-socket\") pod \"router-gateway-2-openshift-default-5c874f788-rjxkq\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:10.493572 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.493503    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a85863-b59f-44f4-9d33-de426efa8221-istio-envoy\") pod \"router-gateway-2-openshift-default-5c874f788-rjxkq\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:10.493572 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.493543    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a85863-b59f-44f4-9d33-de426efa8221-istio-data\") pod \"router-gateway-2-openshift-default-5c874f788-rjxkq\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:10.494043 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.493898    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a85863-b59f-44f4-9d33-de426efa8221-workload-certs\") pod \"router-gateway-2-openshift-default-5c874f788-rjxkq\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:10.494043 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.493921    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a85863-b59f-44f4-9d33-de426efa8221-istio-data\") pod \"router-gateway-2-openshift-default-5c874f788-rjxkq\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:10.494212 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.494192    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a85863-b59f-44f4-9d33-de426efa8221-credential-socket\") pod \"router-gateway-2-openshift-default-5c874f788-rjxkq\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:10.494608 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.494577    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a85863-b59f-44f4-9d33-de426efa8221-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-5c874f788-rjxkq\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:10.494735 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.494713    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a85863-b59f-44f4-9d33-de426efa8221-workload-socket\") pod \"router-gateway-2-openshift-default-5c874f788-rjxkq\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:10.496110 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.496084    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a85863-b59f-44f4-9d33-de426efa8221-istio-envoy\") pod \"router-gateway-2-openshift-default-5c874f788-rjxkq\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:10.496271 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.496250    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a85863-b59f-44f4-9d33-de426efa8221-istio-podinfo\") pod \"router-gateway-2-openshift-default-5c874f788-rjxkq\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:10.502674 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.502646    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a85863-b59f-44f4-9d33-de426efa8221-istio-token\") pod \"router-gateway-2-openshift-default-5c874f788-rjxkq\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:10.502971 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.502931    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2m9g\" (UniqueName: \"kubernetes.io/projected/e0a85863-b59f-44f4-9d33-de426efa8221-kube-api-access-t2m9g\") pod \"router-gateway-2-openshift-default-5c874f788-rjxkq\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:10.573516 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.573480    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:10.862009 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.861845    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"]
Sep 16 18:38:10.864266 ip-10-0-136-122 kubenswrapper[2577]: W0916 18:38:10.864221    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a85863_b59f_44f4_9d33_de426efa8221.slice/crio-59f98a6b795170e9cc32d2e07c03897926538beb9a58154e80ddd14ba470fe45 WatchSource:0}: Error finding container 59f98a6b795170e9cc32d2e07c03897926538beb9a58154e80ddd14ba470fe45: Status 404 returned error can't find the container with id 59f98a6b795170e9cc32d2e07c03897926538beb9a58154e80ddd14ba470fe45
Sep 16 18:38:10.866361 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.866324    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:38:10.866457 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.866392    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:38:10.866457 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:10.866418    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:38:11.099753 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:11.099711    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq" event={"ID":"e0a85863-b59f-44f4-9d33-de426efa8221","Type":"ContainerStarted","Data":"c8a85f6d85e9fb46f0e3f0183056d10fe063baa410a88e602db37928a3c82866"}
Sep 16 18:38:11.099984 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:11.099766    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq" event={"ID":"e0a85863-b59f-44f4-9d33-de426efa8221","Type":"ContainerStarted","Data":"59f98a6b795170e9cc32d2e07c03897926538beb9a58154e80ddd14ba470fe45"}
Sep 16 18:38:11.122278 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:11.122165    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq" podStartSLOduration=1.122149647 podStartE2EDuration="1.122149647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:38:11.120439443 +0000 UTC m=+4412.483118230" watchObservedRunningTime="2026-09-16 18:38:11.122149647 +0000 UTC m=+4412.484828477"
Sep 16 18:38:11.574261 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:11.574226    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:11.575757 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:11.575715    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq" podUID="e0a85863-b59f-44f4-9d33-de426efa8221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Sep 16 18:38:12.578311 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:12.578278    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:13.111782 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:13.111746    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:13.112743 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:13.112717    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:13.481545 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:13.481503    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:17.923459 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:17.923418    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:19.099724 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:19.099685    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q" podUID="8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 16 18:38:22.169558 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:22.169525    2577 generic.go:356] "Generic (PLEG): container finished" podID="e0a85863-b59f-44f4-9d33-de426efa8221" containerID="c8a85f6d85e9fb46f0e3f0183056d10fe063baa410a88e602db37928a3c82866" exitCode=0
Sep 16 18:38:22.170092 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:22.169581    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq" event={"ID":"e0a85863-b59f-44f4-9d33-de426efa8221","Type":"ContainerDied","Data":"c8a85f6d85e9fb46f0e3f0183056d10fe063baa410a88e602db37928a3c82866"}
Sep 16 18:38:22.170092 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:22.169979    2577 scope.go:117] "RemoveContainer" containerID="c8a85f6d85e9fb46f0e3f0183056d10fe063baa410a88e602db37928a3c82866"
Sep 16 18:38:22.170894 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:22.170848    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:38:22.170952 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:22.170933    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:38:22.170992 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:22.170970    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:38:23.177694 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:23.177647    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq" event={"ID":"e0a85863-b59f-44f4-9d33-de426efa8221","Type":"ContainerStarted","Data":"a7be76948978b065265cce9296616b1d72c58e0fe50208a4fe436f7d89b89889"}
Sep 16 18:38:23.574515 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:23.574472    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:23.575934 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:23.575903    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq" podUID="e0a85863-b59f-44f4-9d33-de426efa8221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Sep 16 18:38:24.054548 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:24.054442    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:24.154722 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:24.154684    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:24.574502 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:24.574461    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq" podUID="e0a85863-b59f-44f4-9d33-de426efa8221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Sep 16 18:38:25.574279 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:25.574229    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq" podUID="e0a85863-b59f-44f4-9d33-de426efa8221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Sep 16 18:38:26.575007 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:26.574954    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq" podUID="e0a85863-b59f-44f4-9d33-de426efa8221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Sep 16 18:38:27.102309 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.102267    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"]
Sep 16 18:38:27.111885 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.106837    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"
Sep 16 18:38:27.111885 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.111751    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 16 18:38:27.114439 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.112157    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-26jvd\""
Sep 16 18:38:27.114439 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.112388    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 16 18:38:27.114439 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.112603    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:38:27.124481 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.124452    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"]
Sep 16 18:38:27.139683 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.139599    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:27.143721 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.143693    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7a2aab5-126a-48e1-8535-121e6549b578-home\") pod \"router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z\" (UID: \"a7a2aab5-126a-48e1-8535-121e6549b578\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"
Sep 16 18:38:27.143893 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.143755    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7a2aab5-126a-48e1-8535-121e6549b578-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z\" (UID: \"a7a2aab5-126a-48e1-8535-121e6549b578\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"
Sep 16 18:38:27.143893 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.143785    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq6bl\" (UniqueName: \"kubernetes.io/projected/a7a2aab5-126a-48e1-8535-121e6549b578-kube-api-access-gq6bl\") pod \"router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z\" (UID: \"a7a2aab5-126a-48e1-8535-121e6549b578\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"
Sep 16 18:38:27.143893 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.143838    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7a2aab5-126a-48e1-8535-121e6549b578-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z\" (UID: \"a7a2aab5-126a-48e1-8535-121e6549b578\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"
Sep 16 18:38:27.143893 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.143883    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a2aab5-126a-48e1-8535-121e6549b578-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z\" (UID: \"a7a2aab5-126a-48e1-8535-121e6549b578\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"
Sep 16 18:38:27.144131 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.143909    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7a2aab5-126a-48e1-8535-121e6549b578-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z\" (UID: \"a7a2aab5-126a-48e1-8535-121e6549b578\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"
Sep 16 18:38:27.144131 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.143969    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7a2aab5-126a-48e1-8535-121e6549b578-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z\" (UID: \"a7a2aab5-126a-48e1-8535-121e6549b578\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"
Sep 16 18:38:27.144131 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.144004    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7a2aab5-126a-48e1-8535-121e6549b578-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z\" (UID: \"a7a2aab5-126a-48e1-8535-121e6549b578\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"
Sep 16 18:38:27.245232 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.245189    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7a2aab5-126a-48e1-8535-121e6549b578-home\") pod \"router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z\" (UID: \"a7a2aab5-126a-48e1-8535-121e6549b578\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"
Sep 16 18:38:27.245404 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.245246    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7a2aab5-126a-48e1-8535-121e6549b578-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z\" (UID: \"a7a2aab5-126a-48e1-8535-121e6549b578\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"
Sep 16 18:38:27.245404 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.245264    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq6bl\" (UniqueName: \"kubernetes.io/projected/a7a2aab5-126a-48e1-8535-121e6549b578-kube-api-access-gq6bl\") pod \"router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z\" (UID: \"a7a2aab5-126a-48e1-8535-121e6549b578\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"
Sep 16 18:38:27.245404 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.245292    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7a2aab5-126a-48e1-8535-121e6549b578-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z\" (UID: \"a7a2aab5-126a-48e1-8535-121e6549b578\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"
Sep 16 18:38:27.245404 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.245310    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a2aab5-126a-48e1-8535-121e6549b578-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z\" (UID: \"a7a2aab5-126a-48e1-8535-121e6549b578\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"
Sep 16 18:38:27.245404 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.245325    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7a2aab5-126a-48e1-8535-121e6549b578-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z\" (UID: \"a7a2aab5-126a-48e1-8535-121e6549b578\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"
Sep 16 18:38:27.245404 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.245359    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7a2aab5-126a-48e1-8535-121e6549b578-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z\" (UID: \"a7a2aab5-126a-48e1-8535-121e6549b578\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"
Sep 16 18:38:27.245404 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.245379    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7a2aab5-126a-48e1-8535-121e6549b578-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z\" (UID: \"a7a2aab5-126a-48e1-8535-121e6549b578\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"
Sep 16 18:38:27.246185 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.246151    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7a2aab5-126a-48e1-8535-121e6549b578-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z\" (UID: \"a7a2aab5-126a-48e1-8535-121e6549b578\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"
Sep 16 18:38:27.246397 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.246368    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7a2aab5-126a-48e1-8535-121e6549b578-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z\" (UID: \"a7a2aab5-126a-48e1-8535-121e6549b578\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"
Sep 16 18:38:27.246561 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.246534    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7a2aab5-126a-48e1-8535-121e6549b578-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z\" (UID: \"a7a2aab5-126a-48e1-8535-121e6549b578\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"
Sep 16 18:38:27.246764 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.246720    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7a2aab5-126a-48e1-8535-121e6549b578-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z\" (UID: \"a7a2aab5-126a-48e1-8535-121e6549b578\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"
Sep 16 18:38:27.246901 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.246791    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7a2aab5-126a-48e1-8535-121e6549b578-home\") pod \"router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z\" (UID: \"a7a2aab5-126a-48e1-8535-121e6549b578\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"
Sep 16 18:38:27.247671 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.247645    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7a2aab5-126a-48e1-8535-121e6549b578-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z\" (UID: \"a7a2aab5-126a-48e1-8535-121e6549b578\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"
Sep 16 18:38:27.249483 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.249461    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a2aab5-126a-48e1-8535-121e6549b578-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z\" (UID: \"a7a2aab5-126a-48e1-8535-121e6549b578\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"
Sep 16 18:38:27.259249 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.259220    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq6bl\" (UniqueName: \"kubernetes.io/projected/a7a2aab5-126a-48e1-8535-121e6549b578-kube-api-access-gq6bl\") pod \"router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z\" (UID: \"a7a2aab5-126a-48e1-8535-121e6549b578\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"
Sep 16 18:38:27.426555 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.426453    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"
Sep 16 18:38:27.574675 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.574632    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq" podUID="e0a85863-b59f-44f4-9d33-de426efa8221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Sep 16 18:38:27.751083 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.751057    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"]
Sep 16 18:38:27.753087 ip-10-0-136-122 kubenswrapper[2577]: W0916 18:38:27.753056    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7a2aab5_126a_48e1_8535_121e6549b578.slice/crio-62c37deeeafbcaa99864832e84c0ded276525833a707c34de03169eebadf200c WatchSource:0}: Error finding container 62c37deeeafbcaa99864832e84c0ded276525833a707c34de03169eebadf200c: Status 404 returned error can't find the container with id 62c37deeeafbcaa99864832e84c0ded276525833a707c34de03169eebadf200c
Sep 16 18:38:27.875856 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.875818    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:27.993037 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:27.992915    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:38:28.111964 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:28.111913    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:28.228907 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:28.228570    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z" event={"ID":"a7a2aab5-126a-48e1-8535-121e6549b578","Type":"ContainerStarted","Data":"1a582594524c08bfb7ac7bd0d0d5fb1422e27c43d84ad442cf0810af024cada4"}
Sep 16 18:38:28.228907 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:28.228643    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z" event={"ID":"a7a2aab5-126a-48e1-8535-121e6549b578","Type":"ContainerStarted","Data":"62c37deeeafbcaa99864832e84c0ded276525833a707c34de03169eebadf200c"}
Sep 16 18:38:28.592340 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:28.591512    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:28.593818 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:28.593783    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:38:29.100336 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:29.100286    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q" podUID="8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 16 18:38:31.248422 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:31.248383    2577 generic.go:356] "Generic (PLEG): container finished" podID="a7a2aab5-126a-48e1-8535-121e6549b578" containerID="1a582594524c08bfb7ac7bd0d0d5fb1422e27c43d84ad442cf0810af024cada4" exitCode=0
Sep 16 18:38:31.248837 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:31.248461    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z" event={"ID":"a7a2aab5-126a-48e1-8535-121e6549b578","Type":"ContainerDied","Data":"1a582594524c08bfb7ac7bd0d0d5fb1422e27c43d84ad442cf0810af024cada4"}
Sep 16 18:38:32.257277 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:32.257235    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z" event={"ID":"a7a2aab5-126a-48e1-8535-121e6549b578","Type":"ContainerStarted","Data":"2fe052ae72a87f6e580bd5283c9651b30b5ed05becb700a9c07fff9a4226356b"}
Sep 16 18:38:32.282570 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:32.282511    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z" podStartSLOduration=2.789115545 podStartE2EDuration="5.282490304s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.493374759s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:38:32.281221935 +0000 UTC m=+4433.643900724" watchObservedRunningTime="2026-09-16 18:38:32.282490304 +0000 UTC m=+4433.645169106"
Sep 16 18:38:37.427336 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:37.427280    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"
Sep 16 18:38:37.427852 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:37.427383    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"
Sep 16 18:38:37.429010 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:37.428977    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z" podUID="a7a2aab5-126a-48e1-8535-121e6549b578" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused"
Sep 16 18:38:39.100426 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:39.100382    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q" podUID="8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 16 18:38:47.427922 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:47.427859    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z" podUID="a7a2aab5-126a-48e1-8535-121e6549b578" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused"
Sep 16 18:38:49.099606 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:49.099553    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q" podUID="8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 16 18:38:57.427569 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:57.427525    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z" podUID="a7a2aab5-126a-48e1-8535-121e6549b578" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused"
Sep 16 18:38:59.109849 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:59.109816    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"
Sep 16 18:38:59.118485 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:38:59.118453    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"
Sep 16 18:39:04.522627 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:04.522592    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:04.524115 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:04.524089    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"]
Sep 16 18:39:04.591631 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:04.591568    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:07.428086 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:07.428041    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z" podUID="a7a2aab5-126a-48e1-8535-121e6549b578" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused"
Sep 16 18:39:17.427437 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:17.427389    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z" podUID="a7a2aab5-126a-48e1-8535-121e6549b578" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused"
Sep 16 18:39:19.548183 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:19.548121    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q" podUID="8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68" containerName="main" containerID="cri-o://a31c2ea775ea74988170a58f30052a70f2c1f174b386e6e72aaaa43bbad5529c" gracePeriod=45
Sep 16 18:39:20.111411 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.111373    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q" podUID="8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Sep 16 18:39:20.318670 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.318642    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"
Sep 16 18:39:20.460411 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.460310    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-model-cache\") pod \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\" (UID: \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\") "
Sep 16 18:39:20.460411 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.460353    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-cabundle-cert\") pod \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\" (UID: \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\") "
Sep 16 18:39:20.460411 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.460399    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-kserve-provision-location\") pod \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\" (UID: \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\") "
Sep 16 18:39:20.460700 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.460439    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-dshm\") pod \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\" (UID: \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\") "
Sep 16 18:39:20.460700 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.460523    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9kmx\" (UniqueName: \"kubernetes.io/projected/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-kube-api-access-r9kmx\") pod \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\" (UID: \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\") "
Sep 16 18:39:20.460700 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.460569    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-tls-certs\") pod \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\" (UID: \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\") "
Sep 16 18:39:20.460700 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.460599    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-tmp-dir\") pod \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\" (UID: \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\") "
Sep 16 18:39:20.460700 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.460629    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-home\") pod \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\" (UID: \"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68\") "
Sep 16 18:39:20.460700 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.460639    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-model-cache" (OuterVolumeSpecName: "model-cache") pod "8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68" (UID: "8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:39:20.461038 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.460936    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:39:20.461038 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.460945    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68" (UID: "8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:39:20.461513 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.461488    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-home" (OuterVolumeSpecName: "home") pod "8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68" (UID: "8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:39:20.463559 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.463530    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-kube-api-access-r9kmx" (OuterVolumeSpecName: "kube-api-access-r9kmx") pod "8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68" (UID: "8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68"). InnerVolumeSpecName "kube-api-access-r9kmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:39:20.463668 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.463595    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-dshm" (OuterVolumeSpecName: "dshm") pod "8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68" (UID: "8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:39:20.463668 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.463610    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68" (UID: "8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:39:20.480825 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.480793    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68" (UID: "8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:39:20.491747 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.491721    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68" (UID: "8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:39:20.555414 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.555372    2577 generic.go:356] "Generic (PLEG): container finished" podID="8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68" containerID="a31c2ea775ea74988170a58f30052a70f2c1f174b386e6e72aaaa43bbad5529c" exitCode=0
Sep 16 18:39:20.555825 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.555427    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"
Sep 16 18:39:20.555825 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.555422    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q" event={"ID":"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68","Type":"ContainerDied","Data":"a31c2ea775ea74988170a58f30052a70f2c1f174b386e6e72aaaa43bbad5529c"}
Sep 16 18:39:20.555825 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.555568    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q" event={"ID":"8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68","Type":"ContainerDied","Data":"6200f59b6c3bdcccd62932990ecd85416407193ffc1cfcc88c2fc4cee896f8ca"}
Sep 16 18:39:20.555825 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.555603    2577 scope.go:117] "RemoveContainer" containerID="a31c2ea775ea74988170a58f30052a70f2c1f174b386e6e72aaaa43bbad5529c"
Sep 16 18:39:20.561525 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.561494    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9kmx\" (UniqueName: \"kubernetes.io/projected/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-kube-api-access-r9kmx\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:39:20.561525 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.561524    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:39:20.561655 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.561537    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:39:20.561655 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.561550    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:39:20.561655 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.561562    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-cabundle-cert\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:39:20.561655 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.561573    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-kserve-provision-location\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:39:20.561655 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.561585    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:39:20.565672 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.565657    2577 scope.go:117] "RemoveContainer" containerID="bb9b989bab98bc717147d69de42995dc220224f30a4668563797e161ab2461c3"
Sep 16 18:39:20.575596 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.575579    2577 scope.go:117] "RemoveContainer" containerID="a31c2ea775ea74988170a58f30052a70f2c1f174b386e6e72aaaa43bbad5529c"
Sep 16 18:39:20.575857 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:39:20.575841    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a31c2ea775ea74988170a58f30052a70f2c1f174b386e6e72aaaa43bbad5529c\": container with ID starting with a31c2ea775ea74988170a58f30052a70f2c1f174b386e6e72aaaa43bbad5529c not found: ID does not exist" containerID="a31c2ea775ea74988170a58f30052a70f2c1f174b386e6e72aaaa43bbad5529c"
Sep 16 18:39:20.575936 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.575879    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31c2ea775ea74988170a58f30052a70f2c1f174b386e6e72aaaa43bbad5529c"} err="failed to get container status \"a31c2ea775ea74988170a58f30052a70f2c1f174b386e6e72aaaa43bbad5529c\": rpc error: code = NotFound desc = could not find container \"a31c2ea775ea74988170a58f30052a70f2c1f174b386e6e72aaaa43bbad5529c\": container with ID starting with a31c2ea775ea74988170a58f30052a70f2c1f174b386e6e72aaaa43bbad5529c not found: ID does not exist"
Sep 16 18:39:20.575936 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.575896    2577 scope.go:117] "RemoveContainer" containerID="bb9b989bab98bc717147d69de42995dc220224f30a4668563797e161ab2461c3"
Sep 16 18:39:20.576114 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:39:20.576097    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb9b989bab98bc717147d69de42995dc220224f30a4668563797e161ab2461c3\": container with ID starting with bb9b989bab98bc717147d69de42995dc220224f30a4668563797e161ab2461c3 not found: ID does not exist" containerID="bb9b989bab98bc717147d69de42995dc220224f30a4668563797e161ab2461c3"
Sep 16 18:39:20.576155 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.576118    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb9b989bab98bc717147d69de42995dc220224f30a4668563797e161ab2461c3"} err="failed to get container status \"bb9b989bab98bc717147d69de42995dc220224f30a4668563797e161ab2461c3\": rpc error: code = NotFound desc = could not find container \"bb9b989bab98bc717147d69de42995dc220224f30a4668563797e161ab2461c3\": container with ID starting with bb9b989bab98bc717147d69de42995dc220224f30a4668563797e161ab2461c3 not found: ID does not exist"
Sep 16 18:39:20.583764 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.583742    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"]
Sep 16 18:39:20.587771 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:20.587735    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-55d67d44c5-jv87q"]
Sep 16 18:39:21.083148 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:21.083111    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68" path="/var/lib/kubelet/pods/8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68/volumes"
Sep 16 18:39:27.428106 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:27.428058    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z" podUID="a7a2aab5-126a-48e1-8535-121e6549b578" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused"
Sep 16 18:39:37.427905 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:37.427777    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z" podUID="a7a2aab5-126a-48e1-8535-121e6549b578" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused"
Sep 16 18:39:47.427775 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:47.427729    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z" podUID="a7a2aab5-126a-48e1-8535-121e6549b578" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused"
Sep 16 18:39:54.632349 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:54.632307    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:54.742574 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:54.742535    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:39:57.427875 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:39:57.427807    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z" podUID="a7a2aab5-126a-48e1-8535-121e6549b578" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused"
Sep 16 18:40:01.243523 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:01.243486    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:07.275299 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:07.275261    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:07.427134 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:07.427085    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z" podUID="a7a2aab5-126a-48e1-8535-121e6549b578" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused"
Sep 16 18:40:09.274265 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:09.274219    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:12.794758 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:12.794721    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"]
Sep 16 18:40:12.795158 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:12.795123    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68" containerName="storage-initializer"
Sep 16 18:40:12.795158 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:12.795137    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68" containerName="storage-initializer"
Sep 16 18:40:12.795158 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:12.795149    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68" containerName="main"
Sep 16 18:40:12.795158 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:12.795155    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68" containerName="main"
Sep 16 18:40:12.795290 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:12.795213    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a87dce8-bde4-42c5-aa2a-7a2f4c3c2f68" containerName="main"
Sep 16 18:40:12.799517 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:12.799486    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"
Sep 16 18:40:12.802458 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:12.802433    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 16 18:40:12.802596 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:12.802433    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-nv4gx\""
Sep 16 18:40:12.802681 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:12.802653    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 16 18:40:12.802760 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:12.802743    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:40:12.802805 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:12.802781    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 16 18:40:12.803612 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:12.803598    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 16 18:40:12.815636 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:12.815608    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"]
Sep 16 18:40:12.816163 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:12.816136    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:12.960305 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:12.960272    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwtxw\" (UniqueName: \"kubernetes.io/projected/43f2c681-99f2-4f0c-821f-8f204e875eaa-kube-api-access-xwtxw\") pod \"prestop-hook-test-kserve-6fb696fc7-jwn46\" (UID: \"43f2c681-99f2-4f0c-821f-8f204e875eaa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"
Sep 16 18:40:12.960305 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:12.960311    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43f2c681-99f2-4f0c-821f-8f204e875eaa-model-cache\") pod \"prestop-hook-test-kserve-6fb696fc7-jwn46\" (UID: \"43f2c681-99f2-4f0c-821f-8f204e875eaa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"
Sep 16 18:40:12.960528 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:12.960350    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43f2c681-99f2-4f0c-821f-8f204e875eaa-home\") pod \"prestop-hook-test-kserve-6fb696fc7-jwn46\" (UID: \"43f2c681-99f2-4f0c-821f-8f204e875eaa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"
Sep 16 18:40:12.960528 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:12.960436    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43f2c681-99f2-4f0c-821f-8f204e875eaa-dshm\") pod \"prestop-hook-test-kserve-6fb696fc7-jwn46\" (UID: \"43f2c681-99f2-4f0c-821f-8f204e875eaa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"
Sep 16 18:40:12.960528 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:12.960481    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43f2c681-99f2-4f0c-821f-8f204e875eaa-kserve-provision-location\") pod \"prestop-hook-test-kserve-6fb696fc7-jwn46\" (UID: \"43f2c681-99f2-4f0c-821f-8f204e875eaa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"
Sep 16 18:40:12.960633 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:12.960526    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43f2c681-99f2-4f0c-821f-8f204e875eaa-cabundle-cert\") pod \"prestop-hook-test-kserve-6fb696fc7-jwn46\" (UID: \"43f2c681-99f2-4f0c-821f-8f204e875eaa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"
Sep 16 18:40:12.960633 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:12.960582    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43f2c681-99f2-4f0c-821f-8f204e875eaa-tls-certs\") pod \"prestop-hook-test-kserve-6fb696fc7-jwn46\" (UID: \"43f2c681-99f2-4f0c-821f-8f204e875eaa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"
Sep 16 18:40:12.960700 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:12.960640    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43f2c681-99f2-4f0c-821f-8f204e875eaa-tmp-dir\") pod \"prestop-hook-test-kserve-6fb696fc7-jwn46\" (UID: \"43f2c681-99f2-4f0c-821f-8f204e875eaa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"
Sep 16 18:40:13.054421 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:13.054335    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:13.061523 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:13.061488    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43f2c681-99f2-4f0c-821f-8f204e875eaa-home\") pod \"prestop-hook-test-kserve-6fb696fc7-jwn46\" (UID: \"43f2c681-99f2-4f0c-821f-8f204e875eaa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"
Sep 16 18:40:13.061611 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:13.061546    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43f2c681-99f2-4f0c-821f-8f204e875eaa-dshm\") pod \"prestop-hook-test-kserve-6fb696fc7-jwn46\" (UID: \"43f2c681-99f2-4f0c-821f-8f204e875eaa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"
Sep 16 18:40:13.061611 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:13.061572    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43f2c681-99f2-4f0c-821f-8f204e875eaa-kserve-provision-location\") pod \"prestop-hook-test-kserve-6fb696fc7-jwn46\" (UID: \"43f2c681-99f2-4f0c-821f-8f204e875eaa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"
Sep 16 18:40:13.061611 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:13.061604    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43f2c681-99f2-4f0c-821f-8f204e875eaa-cabundle-cert\") pod \"prestop-hook-test-kserve-6fb696fc7-jwn46\" (UID: \"43f2c681-99f2-4f0c-821f-8f204e875eaa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"
Sep 16 18:40:13.061782 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:13.061753    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43f2c681-99f2-4f0c-821f-8f204e875eaa-tls-certs\") pod \"prestop-hook-test-kserve-6fb696fc7-jwn46\" (UID: \"43f2c681-99f2-4f0c-821f-8f204e875eaa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"
Sep 16 18:40:13.061849 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:13.061830    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43f2c681-99f2-4f0c-821f-8f204e875eaa-tmp-dir\") pod \"prestop-hook-test-kserve-6fb696fc7-jwn46\" (UID: \"43f2c681-99f2-4f0c-821f-8f204e875eaa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"
Sep 16 18:40:13.061931 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:13.061905    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwtxw\" (UniqueName: \"kubernetes.io/projected/43f2c681-99f2-4f0c-821f-8f204e875eaa-kube-api-access-xwtxw\") pod \"prestop-hook-test-kserve-6fb696fc7-jwn46\" (UID: \"43f2c681-99f2-4f0c-821f-8f204e875eaa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"
Sep 16 18:40:13.061992 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:13.061944    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43f2c681-99f2-4f0c-821f-8f204e875eaa-model-cache\") pod \"prestop-hook-test-kserve-6fb696fc7-jwn46\" (UID: \"43f2c681-99f2-4f0c-821f-8f204e875eaa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"
Sep 16 18:40:13.062362 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:13.062336    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43f2c681-99f2-4f0c-821f-8f204e875eaa-home\") pod \"prestop-hook-test-kserve-6fb696fc7-jwn46\" (UID: \"43f2c681-99f2-4f0c-821f-8f204e875eaa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"
Sep 16 18:40:13.062494 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:13.062368    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43f2c681-99f2-4f0c-821f-8f204e875eaa-tmp-dir\") pod \"prestop-hook-test-kserve-6fb696fc7-jwn46\" (UID: \"43f2c681-99f2-4f0c-821f-8f204e875eaa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"
Sep 16 18:40:13.062567 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:13.062498    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43f2c681-99f2-4f0c-821f-8f204e875eaa-kserve-provision-location\") pod \"prestop-hook-test-kserve-6fb696fc7-jwn46\" (UID: \"43f2c681-99f2-4f0c-821f-8f204e875eaa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"
Sep 16 18:40:13.062567 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:13.062512    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43f2c681-99f2-4f0c-821f-8f204e875eaa-model-cache\") pod \"prestop-hook-test-kserve-6fb696fc7-jwn46\" (UID: \"43f2c681-99f2-4f0c-821f-8f204e875eaa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"
Sep 16 18:40:13.062734 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:13.062716    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43f2c681-99f2-4f0c-821f-8f204e875eaa-cabundle-cert\") pod \"prestop-hook-test-kserve-6fb696fc7-jwn46\" (UID: \"43f2c681-99f2-4f0c-821f-8f204e875eaa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"
Sep 16 18:40:13.065067 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:13.065037    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43f2c681-99f2-4f0c-821f-8f204e875eaa-tls-certs\") pod \"prestop-hook-test-kserve-6fb696fc7-jwn46\" (UID: \"43f2c681-99f2-4f0c-821f-8f204e875eaa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"
Sep 16 18:40:13.065283 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:13.065267    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43f2c681-99f2-4f0c-821f-8f204e875eaa-dshm\") pod \"prestop-hook-test-kserve-6fb696fc7-jwn46\" (UID: \"43f2c681-99f2-4f0c-821f-8f204e875eaa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"
Sep 16 18:40:13.071919 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:13.071850    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwtxw\" (UniqueName: \"kubernetes.io/projected/43f2c681-99f2-4f0c-821f-8f204e875eaa-kube-api-access-xwtxw\") pod \"prestop-hook-test-kserve-6fb696fc7-jwn46\" (UID: \"43f2c681-99f2-4f0c-821f-8f204e875eaa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"
Sep 16 18:40:13.110479 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:13.110441    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"
Sep 16 18:40:13.403501 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:13.403454    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"]
Sep 16 18:40:13.404031 ip-10-0-136-122 kubenswrapper[2577]: W0916 18:40:13.404002    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f2c681_99f2_4f0c_821f_8f204e875eaa.slice/crio-fe397345341b1df1ad79ea37b56ecd495a514a67c6e3148b9c1d1a5574afb01e WatchSource:0}: Error finding container fe397345341b1df1ad79ea37b56ecd495a514a67c6e3148b9c1d1a5574afb01e: Status 404 returned error can't find the container with id fe397345341b1df1ad79ea37b56ecd495a514a67c6e3148b9c1d1a5574afb01e
Sep 16 18:40:13.854036 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:13.853992    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46" event={"ID":"43f2c681-99f2-4f0c-821f-8f204e875eaa","Type":"ContainerStarted","Data":"d7036109f16009966b4cf2a119d6f10652cbb7b503f5e00168f795d6f8be1d56"}
Sep 16 18:40:13.854437 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:13.854053    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46" event={"ID":"43f2c681-99f2-4f0c-821f-8f204e875eaa","Type":"ContainerStarted","Data":"fe397345341b1df1ad79ea37b56ecd495a514a67c6e3148b9c1d1a5574afb01e"}
Sep 16 18:40:16.874241 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:16.874202    2577 generic.go:356] "Generic (PLEG): container finished" podID="43f2c681-99f2-4f0c-821f-8f204e875eaa" containerID="d7036109f16009966b4cf2a119d6f10652cbb7b503f5e00168f795d6f8be1d56" exitCode=0
Sep 16 18:40:16.874635 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:16.874292    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46" event={"ID":"43f2c681-99f2-4f0c-821f-8f204e875eaa","Type":"ContainerDied","Data":"d7036109f16009966b4cf2a119d6f10652cbb7b503f5e00168f795d6f8be1d56"}
Sep 16 18:40:17.427328 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:17.427288    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z" podUID="a7a2aab5-126a-48e1-8535-121e6549b578" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": dial tcp 10.133.0.94:8000: connect: connection refused"
Sep 16 18:40:17.883092 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:17.883045    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46" event={"ID":"43f2c681-99f2-4f0c-821f-8f204e875eaa","Type":"ContainerStarted","Data":"415388a1b30fa15769edfd0a4b1764b312327a3576977aa983ffd9966fed37e6"}
Sep 16 18:40:17.909756 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:17.909651    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46" podStartSLOduration=3.483086536 podStartE2EDuration="5.909628831s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.426542295s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:17.907983423 +0000 UTC m=+4539.270662203" watchObservedRunningTime="2026-09-16 18:40:17.909628831 +0000 UTC m=+4539.272307619"
Sep 16 18:40:18.110610 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:18.110571    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"
Sep 16 18:40:23.111209 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:23.111169    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"
Sep 16 18:40:23.113013 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:23.112980    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46" podUID="43f2c681-99f2-4f0c-821f-8f204e875eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": dial tcp 10.133.0.95:8000: connect: connection refused"
Sep 16 18:40:27.437752 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:27.437717    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"
Sep 16 18:40:27.446127 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:27.446057    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"
Sep 16 18:40:32.990711 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:32.990670    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:33.111120 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:33.111072    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46" podUID="43f2c681-99f2-4f0c-821f-8f204e875eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": dial tcp 10.133.0.95:8000: connect: connection refused"
Sep 16 18:40:43.111684 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:43.111642    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46" podUID="43f2c681-99f2-4f0c-821f-8f204e875eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": dial tcp 10.133.0.95:8000: connect: connection refused"
Sep 16 18:40:43.751993 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:43.751955    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:43.780601 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:43.780569    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"]
Sep 16 18:40:43.849845 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:43.849810    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:50.055042 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:50.054992    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:50.156032 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:50.155995    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:53.111634 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:53.111598    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46" podUID="43f2c681-99f2-4f0c-821f-8f204e875eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": dial tcp 10.133.0.95:8000: connect: connection refused"
Sep 16 18:40:58.805252 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:58.805183    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z" podUID="a7a2aab5-126a-48e1-8535-121e6549b578" containerName="main" containerID="cri-o://2fe052ae72a87f6e580bd5283c9651b30b5ed05becb700a9c07fff9a4226356b" gracePeriod=45
Sep 16 18:40:59.559707 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:59.559680    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"
Sep 16 18:40:59.690580 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:59.690492    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7a2aab5-126a-48e1-8535-121e6549b578-kserve-provision-location\") pod \"a7a2aab5-126a-48e1-8535-121e6549b578\" (UID: \"a7a2aab5-126a-48e1-8535-121e6549b578\") "
Sep 16 18:40:59.690580 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:59.690544    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7a2aab5-126a-48e1-8535-121e6549b578-home\") pod \"a7a2aab5-126a-48e1-8535-121e6549b578\" (UID: \"a7a2aab5-126a-48e1-8535-121e6549b578\") "
Sep 16 18:40:59.690580 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:59.690575    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gq6bl\" (UniqueName: \"kubernetes.io/projected/a7a2aab5-126a-48e1-8535-121e6549b578-kube-api-access-gq6bl\") pod \"a7a2aab5-126a-48e1-8535-121e6549b578\" (UID: \"a7a2aab5-126a-48e1-8535-121e6549b578\") "
Sep 16 18:40:59.690879 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:59.690610    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7a2aab5-126a-48e1-8535-121e6549b578-tmp-dir\") pod \"a7a2aab5-126a-48e1-8535-121e6549b578\" (UID: \"a7a2aab5-126a-48e1-8535-121e6549b578\") "
Sep 16 18:40:59.690879 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:59.690629    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7a2aab5-126a-48e1-8535-121e6549b578-model-cache\") pod \"a7a2aab5-126a-48e1-8535-121e6549b578\" (UID: \"a7a2aab5-126a-48e1-8535-121e6549b578\") "
Sep 16 18:40:59.690879 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:59.690684    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7a2aab5-126a-48e1-8535-121e6549b578-cabundle-cert\") pod \"a7a2aab5-126a-48e1-8535-121e6549b578\" (UID: \"a7a2aab5-126a-48e1-8535-121e6549b578\") "
Sep 16 18:40:59.690879 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:59.690728    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a2aab5-126a-48e1-8535-121e6549b578-tls-certs\") pod \"a7a2aab5-126a-48e1-8535-121e6549b578\" (UID: \"a7a2aab5-126a-48e1-8535-121e6549b578\") "
Sep 16 18:40:59.691103 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:59.690919    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a2aab5-126a-48e1-8535-121e6549b578-model-cache" (OuterVolumeSpecName: "model-cache") pod "a7a2aab5-126a-48e1-8535-121e6549b578" (UID: "a7a2aab5-126a-48e1-8535-121e6549b578"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:40:59.691168 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:59.691127    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7a2aab5-126a-48e1-8535-121e6549b578-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a7a2aab5-126a-48e1-8535-121e6549b578" (UID: "a7a2aab5-126a-48e1-8535-121e6549b578"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:40:59.691261 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:59.691230    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7a2aab5-126a-48e1-8535-121e6549b578-dshm\") pod \"a7a2aab5-126a-48e1-8535-121e6549b578\" (UID: \"a7a2aab5-126a-48e1-8535-121e6549b578\") "
Sep 16 18:40:59.691261 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:59.691236    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a2aab5-126a-48e1-8535-121e6549b578-home" (OuterVolumeSpecName: "home") pod "a7a2aab5-126a-48e1-8535-121e6549b578" (UID: "a7a2aab5-126a-48e1-8535-121e6549b578"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:40:59.691571 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:59.691550    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7a2aab5-126a-48e1-8535-121e6549b578-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:40:59.691661 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:59.691576    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7a2aab5-126a-48e1-8535-121e6549b578-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:40:59.691661 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:59.691594    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7a2aab5-126a-48e1-8535-121e6549b578-cabundle-cert\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:40:59.693285 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:59.693258    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a2aab5-126a-48e1-8535-121e6549b578-kube-api-access-gq6bl" (OuterVolumeSpecName: "kube-api-access-gq6bl") pod "a7a2aab5-126a-48e1-8535-121e6549b578" (UID: "a7a2aab5-126a-48e1-8535-121e6549b578"). InnerVolumeSpecName "kube-api-access-gq6bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:40:59.693550 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:59.693530    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7a2aab5-126a-48e1-8535-121e6549b578-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a7a2aab5-126a-48e1-8535-121e6549b578" (UID: "a7a2aab5-126a-48e1-8535-121e6549b578"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:40:59.693795 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:59.693778    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a2aab5-126a-48e1-8535-121e6549b578-dshm" (OuterVolumeSpecName: "dshm") pod "a7a2aab5-126a-48e1-8535-121e6549b578" (UID: "a7a2aab5-126a-48e1-8535-121e6549b578"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:40:59.706357 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:59.706320    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a2aab5-126a-48e1-8535-121e6549b578-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a7a2aab5-126a-48e1-8535-121e6549b578" (UID: "a7a2aab5-126a-48e1-8535-121e6549b578"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:40:59.714914 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:59.714885    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a2aab5-126a-48e1-8535-121e6549b578-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a7a2aab5-126a-48e1-8535-121e6549b578" (UID: "a7a2aab5-126a-48e1-8535-121e6549b578"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:40:59.792790 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:59.792749    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq6bl\" (UniqueName: \"kubernetes.io/projected/a7a2aab5-126a-48e1-8535-121e6549b578-kube-api-access-gq6bl\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:40:59.792970 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:59.792807    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7a2aab5-126a-48e1-8535-121e6549b578-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:40:59.792970 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:59.792820    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a2aab5-126a-48e1-8535-121e6549b578-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:40:59.792970 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:59.792829    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7a2aab5-126a-48e1-8535-121e6549b578-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:40:59.792970 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:40:59.792837    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7a2aab5-126a-48e1-8535-121e6549b578-kserve-provision-location\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:41:00.141990 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:00.141950    2577 generic.go:356] "Generic (PLEG): container finished" podID="a7a2aab5-126a-48e1-8535-121e6549b578" containerID="2fe052ae72a87f6e580bd5283c9651b30b5ed05becb700a9c07fff9a4226356b" exitCode=0
Sep 16 18:41:00.142466 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:00.142000    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"
Sep 16 18:41:00.142466 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:00.142028    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z" event={"ID":"a7a2aab5-126a-48e1-8535-121e6549b578","Type":"ContainerDied","Data":"2fe052ae72a87f6e580bd5283c9651b30b5ed05becb700a9c07fff9a4226356b"}
Sep 16 18:41:00.142466 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:00.142089    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z" event={"ID":"a7a2aab5-126a-48e1-8535-121e6549b578","Type":"ContainerDied","Data":"62c37deeeafbcaa99864832e84c0ded276525833a707c34de03169eebadf200c"}
Sep 16 18:41:00.142466 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:00.142113    2577 scope.go:117] "RemoveContainer" containerID="2fe052ae72a87f6e580bd5283c9651b30b5ed05becb700a9c07fff9a4226356b"
Sep 16 18:41:00.151886 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:00.151848    2577 scope.go:117] "RemoveContainer" containerID="1a582594524c08bfb7ac7bd0d0d5fb1422e27c43d84ad442cf0810af024cada4"
Sep 16 18:41:00.160395 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:00.160375    2577 scope.go:117] "RemoveContainer" containerID="2fe052ae72a87f6e580bd5283c9651b30b5ed05becb700a9c07fff9a4226356b"
Sep 16 18:41:00.160668 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:41:00.160651    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fe052ae72a87f6e580bd5283c9651b30b5ed05becb700a9c07fff9a4226356b\": container with ID starting with 2fe052ae72a87f6e580bd5283c9651b30b5ed05becb700a9c07fff9a4226356b not found: ID does not exist" containerID="2fe052ae72a87f6e580bd5283c9651b30b5ed05becb700a9c07fff9a4226356b"
Sep 16 18:41:00.160720 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:00.160677    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fe052ae72a87f6e580bd5283c9651b30b5ed05becb700a9c07fff9a4226356b"} err="failed to get container status \"2fe052ae72a87f6e580bd5283c9651b30b5ed05becb700a9c07fff9a4226356b\": rpc error: code = NotFound desc = could not find container \"2fe052ae72a87f6e580bd5283c9651b30b5ed05becb700a9c07fff9a4226356b\": container with ID starting with 2fe052ae72a87f6e580bd5283c9651b30b5ed05becb700a9c07fff9a4226356b not found: ID does not exist"
Sep 16 18:41:00.160720 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:00.160694    2577 scope.go:117] "RemoveContainer" containerID="1a582594524c08bfb7ac7bd0d0d5fb1422e27c43d84ad442cf0810af024cada4"
Sep 16 18:41:00.161028 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:41:00.161003    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a582594524c08bfb7ac7bd0d0d5fb1422e27c43d84ad442cf0810af024cada4\": container with ID starting with 1a582594524c08bfb7ac7bd0d0d5fb1422e27c43d84ad442cf0810af024cada4 not found: ID does not exist" containerID="1a582594524c08bfb7ac7bd0d0d5fb1422e27c43d84ad442cf0810af024cada4"
Sep 16 18:41:00.161079 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:00.161038    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a582594524c08bfb7ac7bd0d0d5fb1422e27c43d84ad442cf0810af024cada4"} err="failed to get container status \"1a582594524c08bfb7ac7bd0d0d5fb1422e27c43d84ad442cf0810af024cada4\": rpc error: code = NotFound desc = could not find container \"1a582594524c08bfb7ac7bd0d0d5fb1422e27c43d84ad442cf0810af024cada4\": container with ID starting with 1a582594524c08bfb7ac7bd0d0d5fb1422e27c43d84ad442cf0810af024cada4 not found: ID does not exist"
Sep 16 18:41:00.169970 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:00.169942    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"]
Sep 16 18:41:00.178015 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:00.177991    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-679fb9b895-6sn7z"]
Sep 16 18:41:01.082623 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:01.082579    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7a2aab5-126a-48e1-8535-121e6549b578" path="/var/lib/kubelet/pods/a7a2aab5-126a-48e1-8535-121e6549b578/volumes"
Sep 16 18:41:03.111803 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:03.111758    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46" podUID="43f2c681-99f2-4f0c-821f-8f204e875eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": dial tcp 10.133.0.95:8000: connect: connection refused"
Sep 16 18:41:13.111456 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:13.111409    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46" podUID="43f2c681-99f2-4f0c-821f-8f204e875eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": dial tcp 10.133.0.95:8000: connect: connection refused"
Sep 16 18:41:23.111050 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:23.111006    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46" podUID="43f2c681-99f2-4f0c-821f-8f204e875eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": dial tcp 10.133.0.95:8000: connect: connection refused"
Sep 16 18:41:33.111666 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:33.111618    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46" podUID="43f2c681-99f2-4f0c-821f-8f204e875eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": dial tcp 10.133.0.95:8000: connect: connection refused"
Sep 16 18:41:33.819849 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:33.819815    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:34.038071 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:34.038035    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:38.616122 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:38.616080    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"]
Sep 16 18:41:38.616653 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:38.616318    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq" podUID="e0a85863-b59f-44f4-9d33-de426efa8221" containerName="istio-proxy" containerID="cri-o://a7be76948978b065265cce9296616b1d72c58e0fe50208a4fe436f7d89b89889" gracePeriod=30
Sep 16 18:41:38.713974 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:38.713932    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:41.486287 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:41.486242    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc"]
Sep 16 18:41:41.486773 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:41.486619    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7a2aab5-126a-48e1-8535-121e6549b578" containerName="storage-initializer"
Sep 16 18:41:41.486773 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:41.486632    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7a2aab5-126a-48e1-8535-121e6549b578" containerName="storage-initializer"
Sep 16 18:41:41.486773 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:41.486648    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7a2aab5-126a-48e1-8535-121e6549b578" containerName="main"
Sep 16 18:41:41.486773 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:41.486654    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7a2aab5-126a-48e1-8535-121e6549b578" containerName="main"
Sep 16 18:41:41.486773 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:41.486716    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7a2aab5-126a-48e1-8535-121e6549b578" containerName="main"
Sep 16 18:41:41.491058 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:41.491032    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc"
Sep 16 18:41:41.498424 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:41.497837    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-7wmck\""
Sep 16 18:41:41.498424 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:41.497901    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 16 18:41:41.498424 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:41.497837    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 16 18:41:41.498424 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:41.497837    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 16 18:41:41.504972 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:41.504944    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc"]
Sep 16 18:41:41.518408 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:41.518376    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:41.571189 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:41.571146    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ecd4d0e-8799-446d-afce-23696a05dfe9-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4pfhc\" (UID: \"6ecd4d0e-8799-446d-afce-23696a05dfe9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc"
Sep 16 18:41:41.571189 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:41.571183    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jlcl\" (UniqueName: \"kubernetes.io/projected/6ecd4d0e-8799-446d-afce-23696a05dfe9-kube-api-access-4jlcl\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4pfhc\" (UID: \"6ecd4d0e-8799-446d-afce-23696a05dfe9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc"
Sep 16 18:41:41.571470 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:41.571203    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ecd4d0e-8799-446d-afce-23696a05dfe9-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4pfhc\" (UID: \"6ecd4d0e-8799-446d-afce-23696a05dfe9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc"
Sep 16 18:41:41.571470 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:41.571340    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ecd4d0e-8799-446d-afce-23696a05dfe9-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4pfhc\" (UID: \"6ecd4d0e-8799-446d-afce-23696a05dfe9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc"
Sep 16 18:41:41.571470 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:41.571377    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ecd4d0e-8799-446d-afce-23696a05dfe9-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4pfhc\" (UID: \"6ecd4d0e-8799-446d-afce-23696a05dfe9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc"
Sep 16 18:41:41.571470 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:41.571397    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ecd4d0e-8799-446d-afce-23696a05dfe9-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4pfhc\" (UID: \"6ecd4d0e-8799-446d-afce-23696a05dfe9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc"
Sep 16 18:41:41.671907 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:41.671844    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ecd4d0e-8799-446d-afce-23696a05dfe9-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4pfhc\" (UID: \"6ecd4d0e-8799-446d-afce-23696a05dfe9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc"
Sep 16 18:41:41.671907 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:41.671905    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ecd4d0e-8799-446d-afce-23696a05dfe9-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4pfhc\" (UID: \"6ecd4d0e-8799-446d-afce-23696a05dfe9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc"
Sep 16 18:41:41.672181 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:41.671923    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ecd4d0e-8799-446d-afce-23696a05dfe9-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4pfhc\" (UID: \"6ecd4d0e-8799-446d-afce-23696a05dfe9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc"
Sep 16 18:41:41.672181 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:41.671960    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ecd4d0e-8799-446d-afce-23696a05dfe9-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4pfhc\" (UID: \"6ecd4d0e-8799-446d-afce-23696a05dfe9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc"
Sep 16 18:41:41.672181 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:41.671975    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jlcl\" (UniqueName: \"kubernetes.io/projected/6ecd4d0e-8799-446d-afce-23696a05dfe9-kube-api-access-4jlcl\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4pfhc\" (UID: \"6ecd4d0e-8799-446d-afce-23696a05dfe9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc"
Sep 16 18:41:41.672181 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:41.671992    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ecd4d0e-8799-446d-afce-23696a05dfe9-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4pfhc\" (UID: \"6ecd4d0e-8799-446d-afce-23696a05dfe9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc"
Sep 16 18:41:41.672467 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:41.672317    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ecd4d0e-8799-446d-afce-23696a05dfe9-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4pfhc\" (UID: \"6ecd4d0e-8799-446d-afce-23696a05dfe9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc"
Sep 16 18:41:41.672467 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:41.672362    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ecd4d0e-8799-446d-afce-23696a05dfe9-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4pfhc\" (UID: \"6ecd4d0e-8799-446d-afce-23696a05dfe9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc"
Sep 16 18:41:41.672590 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:41.672465    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ecd4d0e-8799-446d-afce-23696a05dfe9-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4pfhc\" (UID: \"6ecd4d0e-8799-446d-afce-23696a05dfe9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc"
Sep 16 18:41:41.674954 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:41.674929    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ecd4d0e-8799-446d-afce-23696a05dfe9-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4pfhc\" (UID: \"6ecd4d0e-8799-446d-afce-23696a05dfe9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc"
Sep 16 18:41:41.675145 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:41.675127    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ecd4d0e-8799-446d-afce-23696a05dfe9-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4pfhc\" (UID: \"6ecd4d0e-8799-446d-afce-23696a05dfe9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc"
Sep 16 18:41:41.684626 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:41.684596    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jlcl\" (UniqueName: \"kubernetes.io/projected/6ecd4d0e-8799-446d-afce-23696a05dfe9-kube-api-access-4jlcl\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4pfhc\" (UID: \"6ecd4d0e-8799-446d-afce-23696a05dfe9\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc"
Sep 16 18:41:41.769609 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:41.769497    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:41.802355 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:41.802318    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc"
Sep 16 18:41:42.086561 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:42.086531    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc"]
Sep 16 18:41:42.088294 ip-10-0-136-122 kubenswrapper[2577]: W0916 18:41:42.088266    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ecd4d0e_8799_446d_afce_23696a05dfe9.slice/crio-1d5fde62a9e2a40a5043cb7c3dd0e52a35fd7e461348ccd379783a6e83e77441 WatchSource:0}: Error finding container 1d5fde62a9e2a40a5043cb7c3dd0e52a35fd7e461348ccd379783a6e83e77441: Status 404 returned error can't find the container with id 1d5fde62a9e2a40a5043cb7c3dd0e52a35fd7e461348ccd379783a6e83e77441
Sep 16 18:41:42.401810 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:42.401681    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc" event={"ID":"6ecd4d0e-8799-446d-afce-23696a05dfe9","Type":"ContainerStarted","Data":"12e6639eed518e8f1a6f169d0e204809ae59f96322d7c36ad9d3a94d29e0692a"}
Sep 16 18:41:42.401810 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:42.401782    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc" event={"ID":"6ecd4d0e-8799-446d-afce-23696a05dfe9","Type":"ContainerStarted","Data":"1d5fde62a9e2a40a5043cb7c3dd0e52a35fd7e461348ccd379783a6e83e77441"}
Sep 16 18:41:42.423997 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:42.423940    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc" podStartSLOduration=1.423923985 podStartE2EDuration="1.423923985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:42.42199334 +0000 UTC m=+4623.784672128" watchObservedRunningTime="2026-09-16 18:41:42.423923985 +0000 UTC m=+4623.786602771"
Sep 16 18:41:42.802789 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:42.802688    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc"
Sep 16 18:41:43.111426 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:43.111385    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46" podUID="43f2c681-99f2-4f0c-821f-8f204e875eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": dial tcp 10.133.0.95:8000: connect: connection refused"
Sep 16 18:41:43.594893 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:43.594828    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq" podUID="e0a85863-b59f-44f4-9d33-de426efa8221" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:41:43.874586 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:43.874559    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:41:43.995282 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:43.995244    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a85863-b59f-44f4-9d33-de426efa8221-istio-podinfo\") pod \"e0a85863-b59f-44f4-9d33-de426efa8221\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") "
Sep 16 18:41:43.995472 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:43.995306    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a85863-b59f-44f4-9d33-de426efa8221-credential-socket\") pod \"e0a85863-b59f-44f4-9d33-de426efa8221\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") "
Sep 16 18:41:43.995472 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:43.995389    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a85863-b59f-44f4-9d33-de426efa8221-istio-token\") pod \"e0a85863-b59f-44f4-9d33-de426efa8221\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") "
Sep 16 18:41:43.995472 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:43.995414    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a85863-b59f-44f4-9d33-de426efa8221-istiod-ca-cert\") pod \"e0a85863-b59f-44f4-9d33-de426efa8221\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") "
Sep 16 18:41:43.995472 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:43.995449    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2m9g\" (UniqueName: \"kubernetes.io/projected/e0a85863-b59f-44f4-9d33-de426efa8221-kube-api-access-t2m9g\") pod \"e0a85863-b59f-44f4-9d33-de426efa8221\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") "
Sep 16 18:41:43.995472 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:43.995470    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a85863-b59f-44f4-9d33-de426efa8221-workload-certs\") pod \"e0a85863-b59f-44f4-9d33-de426efa8221\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") "
Sep 16 18:41:43.995743 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:43.995493    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a85863-b59f-44f4-9d33-de426efa8221-istio-envoy\") pod \"e0a85863-b59f-44f4-9d33-de426efa8221\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") "
Sep 16 18:41:43.995743 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:43.995521    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a85863-b59f-44f4-9d33-de426efa8221-istio-data\") pod \"e0a85863-b59f-44f4-9d33-de426efa8221\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") "
Sep 16 18:41:43.995743 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:43.995571    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a85863-b59f-44f4-9d33-de426efa8221-workload-socket\") pod \"e0a85863-b59f-44f4-9d33-de426efa8221\" (UID: \"e0a85863-b59f-44f4-9d33-de426efa8221\") "
Sep 16 18:41:43.995743 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:43.995706    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a85863-b59f-44f4-9d33-de426efa8221-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e0a85863-b59f-44f4-9d33-de426efa8221" (UID: "e0a85863-b59f-44f4-9d33-de426efa8221"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:41:43.996009 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:43.995973    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a85863-b59f-44f4-9d33-de426efa8221-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e0a85863-b59f-44f4-9d33-de426efa8221" (UID: "e0a85863-b59f-44f4-9d33-de426efa8221"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:41:43.996263 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:43.996234    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a85863-b59f-44f4-9d33-de426efa8221-credential-socket\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:41:43.996347 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:43.996317    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a85863-b59f-44f4-9d33-de426efa8221-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e0a85863-b59f-44f4-9d33-de426efa8221" (UID: "e0a85863-b59f-44f4-9d33-de426efa8221"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:41:43.996403 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:43.996350    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0a85863-b59f-44f4-9d33-de426efa8221-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e0a85863-b59f-44f4-9d33-de426efa8221" (UID: "e0a85863-b59f-44f4-9d33-de426efa8221"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:41:43.996646 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:43.996620    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a85863-b59f-44f4-9d33-de426efa8221-istio-data" (OuterVolumeSpecName: "istio-data") pod "e0a85863-b59f-44f4-9d33-de426efa8221" (UID: "e0a85863-b59f-44f4-9d33-de426efa8221"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:41:43.998086 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:43.998057    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0a85863-b59f-44f4-9d33-de426efa8221-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e0a85863-b59f-44f4-9d33-de426efa8221" (UID: "e0a85863-b59f-44f4-9d33-de426efa8221"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:41:43.998197 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:43.998145    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a85863-b59f-44f4-9d33-de426efa8221-istio-token" (OuterVolumeSpecName: "istio-token") pod "e0a85863-b59f-44f4-9d33-de426efa8221" (UID: "e0a85863-b59f-44f4-9d33-de426efa8221"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:41:43.998434 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:43.998412    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a85863-b59f-44f4-9d33-de426efa8221-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e0a85863-b59f-44f4-9d33-de426efa8221" (UID: "e0a85863-b59f-44f4-9d33-de426efa8221"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:41:43.998607 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:43.998582    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a85863-b59f-44f4-9d33-de426efa8221-kube-api-access-t2m9g" (OuterVolumeSpecName: "kube-api-access-t2m9g") pod "e0a85863-b59f-44f4-9d33-de426efa8221" (UID: "e0a85863-b59f-44f4-9d33-de426efa8221"). InnerVolumeSpecName "kube-api-access-t2m9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:41:44.097514 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:44.097477    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a85863-b59f-44f4-9d33-de426efa8221-istio-token\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:41:44.097514 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:44.097511    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a85863-b59f-44f4-9d33-de426efa8221-istiod-ca-cert\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:41:44.097514 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:44.097520    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2m9g\" (UniqueName: \"kubernetes.io/projected/e0a85863-b59f-44f4-9d33-de426efa8221-kube-api-access-t2m9g\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:41:44.097773 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:44.097530    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a85863-b59f-44f4-9d33-de426efa8221-workload-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:41:44.097773 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:44.097540    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a85863-b59f-44f4-9d33-de426efa8221-istio-envoy\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:41:44.097773 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:44.097549    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a85863-b59f-44f4-9d33-de426efa8221-istio-data\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:41:44.097773 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:44.097556    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a85863-b59f-44f4-9d33-de426efa8221-workload-socket\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:41:44.097773 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:44.097565    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a85863-b59f-44f4-9d33-de426efa8221-istio-podinfo\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:41:44.417569 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:44.417529    2577 generic.go:356] "Generic (PLEG): container finished" podID="e0a85863-b59f-44f4-9d33-de426efa8221" containerID="a7be76948978b065265cce9296616b1d72c58e0fe50208a4fe436f7d89b89889" exitCode=0
Sep 16 18:41:44.417779 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:44.417592    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"
Sep 16 18:41:44.417779 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:44.417605    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq" event={"ID":"e0a85863-b59f-44f4-9d33-de426efa8221","Type":"ContainerDied","Data":"a7be76948978b065265cce9296616b1d72c58e0fe50208a4fe436f7d89b89889"}
Sep 16 18:41:44.417779 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:44.417671    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq" event={"ID":"e0a85863-b59f-44f4-9d33-de426efa8221","Type":"ContainerDied","Data":"59f98a6b795170e9cc32d2e07c03897926538beb9a58154e80ddd14ba470fe45"}
Sep 16 18:41:44.417779 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:44.417694    2577 scope.go:117] "RemoveContainer" containerID="a7be76948978b065265cce9296616b1d72c58e0fe50208a4fe436f7d89b89889"
Sep 16 18:41:44.428158 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:44.428140    2577 scope.go:117] "RemoveContainer" containerID="c8a85f6d85e9fb46f0e3f0183056d10fe063baa410a88e602db37928a3c82866"
Sep 16 18:41:44.438798 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:44.438778    2577 scope.go:117] "RemoveContainer" containerID="a7be76948978b065265cce9296616b1d72c58e0fe50208a4fe436f7d89b89889"
Sep 16 18:41:44.439152 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:41:44.439126    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7be76948978b065265cce9296616b1d72c58e0fe50208a4fe436f7d89b89889\": container with ID starting with a7be76948978b065265cce9296616b1d72c58e0fe50208a4fe436f7d89b89889 not found: ID does not exist" containerID="a7be76948978b065265cce9296616b1d72c58e0fe50208a4fe436f7d89b89889"
Sep 16 18:41:44.439226 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:44.439161    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7be76948978b065265cce9296616b1d72c58e0fe50208a4fe436f7d89b89889"} err="failed to get container status \"a7be76948978b065265cce9296616b1d72c58e0fe50208a4fe436f7d89b89889\": rpc error: code = NotFound desc = could not find container \"a7be76948978b065265cce9296616b1d72c58e0fe50208a4fe436f7d89b89889\": container with ID starting with a7be76948978b065265cce9296616b1d72c58e0fe50208a4fe436f7d89b89889 not found: ID does not exist"
Sep 16 18:41:44.439226 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:44.439178    2577 scope.go:117] "RemoveContainer" containerID="c8a85f6d85e9fb46f0e3f0183056d10fe063baa410a88e602db37928a3c82866"
Sep 16 18:41:44.439451 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:41:44.439431    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8a85f6d85e9fb46f0e3f0183056d10fe063baa410a88e602db37928a3c82866\": container with ID starting with c8a85f6d85e9fb46f0e3f0183056d10fe063baa410a88e602db37928a3c82866 not found: ID does not exist" containerID="c8a85f6d85e9fb46f0e3f0183056d10fe063baa410a88e602db37928a3c82866"
Sep 16 18:41:44.439520 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:44.439461    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8a85f6d85e9fb46f0e3f0183056d10fe063baa410a88e602db37928a3c82866"} err="failed to get container status \"c8a85f6d85e9fb46f0e3f0183056d10fe063baa410a88e602db37928a3c82866\": rpc error: code = NotFound desc = could not find container \"c8a85f6d85e9fb46f0e3f0183056d10fe063baa410a88e602db37928a3c82866\": container with ID starting with c8a85f6d85e9fb46f0e3f0183056d10fe063baa410a88e602db37928a3c82866 not found: ID does not exist"
Sep 16 18:41:44.445700 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:44.445676    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"]
Sep 16 18:41:44.450674 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:44.450652    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-5c874f788-rjxkq"]
Sep 16 18:41:45.081843 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:45.081805    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0a85863-b59f-44f4-9d33-de426efa8221" path="/var/lib/kubelet/pods/e0a85863-b59f-44f4-9d33-de426efa8221/volumes"
Sep 16 18:41:51.802576 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:51.802526    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc"
Sep 16 18:41:51.815469 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:51.815435    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc"
Sep 16 18:41:52.475303 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:52.475272    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc"
Sep 16 18:41:53.111076 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:53.111035    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46" podUID="43f2c681-99f2-4f0c-821f-8f204e875eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": dial tcp 10.133.0.95:8000: connect: connection refused"
Sep 16 18:41:53.873757 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:41:53.873698    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:00.591972 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:00.591925    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:02.303991 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:02.303896    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:03.111263 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:03.111211    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46" podUID="43f2c681-99f2-4f0c-821f-8f204e875eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": dial tcp 10.133.0.95:8000: connect: connection refused"
Sep 16 18:42:08.854282 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:08.854243    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:13.121010 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:13.120974    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"
Sep 16 18:42:13.129184 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:13.129153    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"
Sep 16 18:42:15.577775 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:15.577725    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"]
Sep 16 18:42:27.085610 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:27.085577    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:30.605895 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:30.605818    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46" podUID="43f2c681-99f2-4f0c-821f-8f204e875eaa" containerName="main" containerID="cri-o://415388a1b30fa15769edfd0a4b1764b312327a3576977aa983ffd9966fed37e6" gracePeriod=45
Sep 16 18:42:30.723404 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:30.723366    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc"]
Sep 16 18:42:30.732429 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:42:30.732382    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 18:42:30.732429 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:42:30.732429 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:42:30.732429 ip-10-0-136-122 kubenswrapper[2577]:  > containerID="12e6639eed518e8f1a6f169d0e204809ae59f96322d7c36ad9d3a94d29e0692a" cmd=["/bin/sleep","15"]
Sep 16 18:42:30.732734 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:30.732442    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc" message="" err=<
Sep 16 18:42:30.732734 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:42:30.732734 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:42:30.732734 ip-10-0-136-122 kubenswrapper[2577]:  >
Sep 16 18:42:30.732734 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:42:30.732454    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 18:42:30.732734 ip-10-0-136-122 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:42:30.732734 ip-10-0-136-122 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 16 18:42:30.732734 ip-10-0-136-122 kubenswrapper[2577]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc" podUID="6ecd4d0e-8799-446d-afce-23696a05dfe9" containerName="main" containerID="cri-o://12e6639eed518e8f1a6f169d0e204809ae59f96322d7c36ad9d3a94d29e0692a"
Sep 16 18:42:30.732734 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:30.732491    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc" podUID="6ecd4d0e-8799-446d-afce-23696a05dfe9" containerName="main" containerID="cri-o://12e6639eed518e8f1a6f169d0e204809ae59f96322d7c36ad9d3a94d29e0692a" gracePeriod=60
Sep 16 18:42:31.092569 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.092540    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc"
Sep 16 18:42:31.121979 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.121889    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46" podUID="43f2c681-99f2-4f0c-821f-8f204e875eaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": dial tcp 10.133.0.95:8000: connect: connection refused"
Sep 16 18:42:31.218924 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.218825    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ecd4d0e-8799-446d-afce-23696a05dfe9-home\") pod \"6ecd4d0e-8799-446d-afce-23696a05dfe9\" (UID: \"6ecd4d0e-8799-446d-afce-23696a05dfe9\") "
Sep 16 18:42:31.219095 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.218957    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ecd4d0e-8799-446d-afce-23696a05dfe9-model-cache\") pod \"6ecd4d0e-8799-446d-afce-23696a05dfe9\" (UID: \"6ecd4d0e-8799-446d-afce-23696a05dfe9\") "
Sep 16 18:42:31.219095 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.219003    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jlcl\" (UniqueName: \"kubernetes.io/projected/6ecd4d0e-8799-446d-afce-23696a05dfe9-kube-api-access-4jlcl\") pod \"6ecd4d0e-8799-446d-afce-23696a05dfe9\" (UID: \"6ecd4d0e-8799-446d-afce-23696a05dfe9\") "
Sep 16 18:42:31.219095 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.219028    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ecd4d0e-8799-446d-afce-23696a05dfe9-tmp-dir\") pod \"6ecd4d0e-8799-446d-afce-23696a05dfe9\" (UID: \"6ecd4d0e-8799-446d-afce-23696a05dfe9\") "
Sep 16 18:42:31.219095 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.219059    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ecd4d0e-8799-446d-afce-23696a05dfe9-dshm\") pod \"6ecd4d0e-8799-446d-afce-23696a05dfe9\" (UID: \"6ecd4d0e-8799-446d-afce-23696a05dfe9\") "
Sep 16 18:42:31.219317 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.219090    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ecd4d0e-8799-446d-afce-23696a05dfe9-home" (OuterVolumeSpecName: "home") pod "6ecd4d0e-8799-446d-afce-23696a05dfe9" (UID: "6ecd4d0e-8799-446d-afce-23696a05dfe9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:31.219317 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.219126    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ecd4d0e-8799-446d-afce-23696a05dfe9-tls-certs\") pod \"6ecd4d0e-8799-446d-afce-23696a05dfe9\" (UID: \"6ecd4d0e-8799-446d-afce-23696a05dfe9\") "
Sep 16 18:42:31.219317 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.219254    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ecd4d0e-8799-446d-afce-23696a05dfe9-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ecd4d0e-8799-446d-afce-23696a05dfe9" (UID: "6ecd4d0e-8799-446d-afce-23696a05dfe9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:31.219317 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.219288    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ecd4d0e-8799-446d-afce-23696a05dfe9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ecd4d0e-8799-446d-afce-23696a05dfe9" (UID: "6ecd4d0e-8799-446d-afce-23696a05dfe9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:31.219514 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.219434    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ecd4d0e-8799-446d-afce-23696a05dfe9-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:42:31.219514 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.219474    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ecd4d0e-8799-446d-afce-23696a05dfe9-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:42:31.219514 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.219487    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ecd4d0e-8799-446d-afce-23696a05dfe9-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:42:31.221214 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.221191    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ecd4d0e-8799-446d-afce-23696a05dfe9-dshm" (OuterVolumeSpecName: "dshm") pod "6ecd4d0e-8799-446d-afce-23696a05dfe9" (UID: "6ecd4d0e-8799-446d-afce-23696a05dfe9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:31.221546 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.221525    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ecd4d0e-8799-446d-afce-23696a05dfe9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ecd4d0e-8799-446d-afce-23696a05dfe9" (UID: "6ecd4d0e-8799-446d-afce-23696a05dfe9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:31.221546 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.221534    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ecd4d0e-8799-446d-afce-23696a05dfe9-kube-api-access-4jlcl" (OuterVolumeSpecName: "kube-api-access-4jlcl") pod "6ecd4d0e-8799-446d-afce-23696a05dfe9" (UID: "6ecd4d0e-8799-446d-afce-23696a05dfe9"). InnerVolumeSpecName "kube-api-access-4jlcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:31.320466 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.320430    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ecd4d0e-8799-446d-afce-23696a05dfe9-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:42:31.320466 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.320463    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jlcl\" (UniqueName: \"kubernetes.io/projected/6ecd4d0e-8799-446d-afce-23696a05dfe9-kube-api-access-4jlcl\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:42:31.320466 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.320474    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ecd4d0e-8799-446d-afce-23696a05dfe9-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:42:31.351837 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.351809    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"
Sep 16 18:42:31.522608 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.522512    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43f2c681-99f2-4f0c-821f-8f204e875eaa-kserve-provision-location\") pod \"43f2c681-99f2-4f0c-821f-8f204e875eaa\" (UID: \"43f2c681-99f2-4f0c-821f-8f204e875eaa\") "
Sep 16 18:42:31.522608 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.522570    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwtxw\" (UniqueName: \"kubernetes.io/projected/43f2c681-99f2-4f0c-821f-8f204e875eaa-kube-api-access-xwtxw\") pod \"43f2c681-99f2-4f0c-821f-8f204e875eaa\" (UID: \"43f2c681-99f2-4f0c-821f-8f204e875eaa\") "
Sep 16 18:42:31.522608 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.522606    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43f2c681-99f2-4f0c-821f-8f204e875eaa-home\") pod \"43f2c681-99f2-4f0c-821f-8f204e875eaa\" (UID: \"43f2c681-99f2-4f0c-821f-8f204e875eaa\") "
Sep 16 18:42:31.522935 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.522708    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43f2c681-99f2-4f0c-821f-8f204e875eaa-cabundle-cert\") pod \"43f2c681-99f2-4f0c-821f-8f204e875eaa\" (UID: \"43f2c681-99f2-4f0c-821f-8f204e875eaa\") "
Sep 16 18:42:31.522935 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.522761    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43f2c681-99f2-4f0c-821f-8f204e875eaa-model-cache\") pod \"43f2c681-99f2-4f0c-821f-8f204e875eaa\" (UID: \"43f2c681-99f2-4f0c-821f-8f204e875eaa\") "
Sep 16 18:42:31.522935 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.522802    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43f2c681-99f2-4f0c-821f-8f204e875eaa-tls-certs\") pod \"43f2c681-99f2-4f0c-821f-8f204e875eaa\" (UID: \"43f2c681-99f2-4f0c-821f-8f204e875eaa\") "
Sep 16 18:42:31.522935 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.522858    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43f2c681-99f2-4f0c-821f-8f204e875eaa-dshm\") pod \"43f2c681-99f2-4f0c-821f-8f204e875eaa\" (UID: \"43f2c681-99f2-4f0c-821f-8f204e875eaa\") "
Sep 16 18:42:31.523157 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.522957    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43f2c681-99f2-4f0c-821f-8f204e875eaa-tmp-dir\") pod \"43f2c681-99f2-4f0c-821f-8f204e875eaa\" (UID: \"43f2c681-99f2-4f0c-821f-8f204e875eaa\") "
Sep 16 18:42:31.523157 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.523106    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43f2c681-99f2-4f0c-821f-8f204e875eaa-model-cache" (OuterVolumeSpecName: "model-cache") pod "43f2c681-99f2-4f0c-821f-8f204e875eaa" (UID: "43f2c681-99f2-4f0c-821f-8f204e875eaa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:31.523252 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.523235    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43f2c681-99f2-4f0c-821f-8f204e875eaa-model-cache\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:42:31.523400 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.523362    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43f2c681-99f2-4f0c-821f-8f204e875eaa-home" (OuterVolumeSpecName: "home") pod "43f2c681-99f2-4f0c-821f-8f204e875eaa" (UID: "43f2c681-99f2-4f0c-821f-8f204e875eaa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:31.523521 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.523428    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43f2c681-99f2-4f0c-821f-8f204e875eaa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "43f2c681-99f2-4f0c-821f-8f204e875eaa" (UID: "43f2c681-99f2-4f0c-821f-8f204e875eaa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:31.525525 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.525499    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43f2c681-99f2-4f0c-821f-8f204e875eaa-dshm" (OuterVolumeSpecName: "dshm") pod "43f2c681-99f2-4f0c-821f-8f204e875eaa" (UID: "43f2c681-99f2-4f0c-821f-8f204e875eaa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:31.525649 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.525525    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43f2c681-99f2-4f0c-821f-8f204e875eaa-kube-api-access-xwtxw" (OuterVolumeSpecName: "kube-api-access-xwtxw") pod "43f2c681-99f2-4f0c-821f-8f204e875eaa" (UID: "43f2c681-99f2-4f0c-821f-8f204e875eaa"). InnerVolumeSpecName "kube-api-access-xwtxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:31.525649 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.525613    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43f2c681-99f2-4f0c-821f-8f204e875eaa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "43f2c681-99f2-4f0c-821f-8f204e875eaa" (UID: "43f2c681-99f2-4f0c-821f-8f204e875eaa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:31.536656 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.536616    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43f2c681-99f2-4f0c-821f-8f204e875eaa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "43f2c681-99f2-4f0c-821f-8f204e875eaa" (UID: "43f2c681-99f2-4f0c-821f-8f204e875eaa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:31.545890 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.545847    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43f2c681-99f2-4f0c-821f-8f204e875eaa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "43f2c681-99f2-4f0c-821f-8f204e875eaa" (UID: "43f2c681-99f2-4f0c-821f-8f204e875eaa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:31.624342 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.624305    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43f2c681-99f2-4f0c-821f-8f204e875eaa-dshm\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:42:31.624342 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.624339    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43f2c681-99f2-4f0c-821f-8f204e875eaa-tmp-dir\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:42:31.624342 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.624350    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43f2c681-99f2-4f0c-821f-8f204e875eaa-kserve-provision-location\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:42:31.624788 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.624360    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwtxw\" (UniqueName: \"kubernetes.io/projected/43f2c681-99f2-4f0c-821f-8f204e875eaa-kube-api-access-xwtxw\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:42:31.624788 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.624369    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43f2c681-99f2-4f0c-821f-8f204e875eaa-home\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:42:31.624788 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.624377    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43f2c681-99f2-4f0c-821f-8f204e875eaa-cabundle-cert\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:42:31.624788 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.624385    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43f2c681-99f2-4f0c-821f-8f204e875eaa-tls-certs\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:42:31.680063 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.680029    2577 generic.go:356] "Generic (PLEG): container finished" podID="6ecd4d0e-8799-446d-afce-23696a05dfe9" containerID="12e6639eed518e8f1a6f169d0e204809ae59f96322d7c36ad9d3a94d29e0692a" exitCode=0
Sep 16 18:42:31.680255 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.680075    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc"
Sep 16 18:42:31.680255 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.680111    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc" event={"ID":"6ecd4d0e-8799-446d-afce-23696a05dfe9","Type":"ContainerDied","Data":"12e6639eed518e8f1a6f169d0e204809ae59f96322d7c36ad9d3a94d29e0692a"}
Sep 16 18:42:31.680255 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.680178    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc" event={"ID":"6ecd4d0e-8799-446d-afce-23696a05dfe9","Type":"ContainerDied","Data":"1d5fde62a9e2a40a5043cb7c3dd0e52a35fd7e461348ccd379783a6e83e77441"}
Sep 16 18:42:31.680255 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.680209    2577 scope.go:117] "RemoveContainer" containerID="12e6639eed518e8f1a6f169d0e204809ae59f96322d7c36ad9d3a94d29e0692a"
Sep 16 18:42:31.681999 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.681973    2577 generic.go:356] "Generic (PLEG): container finished" podID="43f2c681-99f2-4f0c-821f-8f204e875eaa" containerID="415388a1b30fa15769edfd0a4b1764b312327a3576977aa983ffd9966fed37e6" exitCode=0
Sep 16 18:42:31.682125 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.682011    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46" event={"ID":"43f2c681-99f2-4f0c-821f-8f204e875eaa","Type":"ContainerDied","Data":"415388a1b30fa15769edfd0a4b1764b312327a3576977aa983ffd9966fed37e6"}
Sep 16 18:42:31.682125 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.682028    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"
Sep 16 18:42:31.682125 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.682068    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46" event={"ID":"43f2c681-99f2-4f0c-821f-8f204e875eaa","Type":"ContainerDied","Data":"fe397345341b1df1ad79ea37b56ecd495a514a67c6e3148b9c1d1a5574afb01e"}
Sep 16 18:42:31.691071 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.691050    2577 scope.go:117] "RemoveContainer" containerID="12e6639eed518e8f1a6f169d0e204809ae59f96322d7c36ad9d3a94d29e0692a"
Sep 16 18:42:31.691422 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:42:31.691392    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12e6639eed518e8f1a6f169d0e204809ae59f96322d7c36ad9d3a94d29e0692a\": container with ID starting with 12e6639eed518e8f1a6f169d0e204809ae59f96322d7c36ad9d3a94d29e0692a not found: ID does not exist" containerID="12e6639eed518e8f1a6f169d0e204809ae59f96322d7c36ad9d3a94d29e0692a"
Sep 16 18:42:31.691526 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.691431    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12e6639eed518e8f1a6f169d0e204809ae59f96322d7c36ad9d3a94d29e0692a"} err="failed to get container status \"12e6639eed518e8f1a6f169d0e204809ae59f96322d7c36ad9d3a94d29e0692a\": rpc error: code = NotFound desc = could not find container \"12e6639eed518e8f1a6f169d0e204809ae59f96322d7c36ad9d3a94d29e0692a\": container with ID starting with 12e6639eed518e8f1a6f169d0e204809ae59f96322d7c36ad9d3a94d29e0692a not found: ID does not exist"
Sep 16 18:42:31.691526 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.691455    2577 scope.go:117] "RemoveContainer" containerID="415388a1b30fa15769edfd0a4b1764b312327a3576977aa983ffd9966fed37e6"
Sep 16 18:42:31.700423 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.700399    2577 scope.go:117] "RemoveContainer" containerID="d7036109f16009966b4cf2a119d6f10652cbb7b503f5e00168f795d6f8be1d56"
Sep 16 18:42:31.708614 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.708595    2577 scope.go:117] "RemoveContainer" containerID="415388a1b30fa15769edfd0a4b1764b312327a3576977aa983ffd9966fed37e6"
Sep 16 18:42:31.708908 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:42:31.708855    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"415388a1b30fa15769edfd0a4b1764b312327a3576977aa983ffd9966fed37e6\": container with ID starting with 415388a1b30fa15769edfd0a4b1764b312327a3576977aa983ffd9966fed37e6 not found: ID does not exist" containerID="415388a1b30fa15769edfd0a4b1764b312327a3576977aa983ffd9966fed37e6"
Sep 16 18:42:31.708908 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.708897    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"415388a1b30fa15769edfd0a4b1764b312327a3576977aa983ffd9966fed37e6"} err="failed to get container status \"415388a1b30fa15769edfd0a4b1764b312327a3576977aa983ffd9966fed37e6\": rpc error: code = NotFound desc = could not find container \"415388a1b30fa15769edfd0a4b1764b312327a3576977aa983ffd9966fed37e6\": container with ID starting with 415388a1b30fa15769edfd0a4b1764b312327a3576977aa983ffd9966fed37e6 not found: ID does not exist"
Sep 16 18:42:31.709064 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.708914    2577 scope.go:117] "RemoveContainer" containerID="d7036109f16009966b4cf2a119d6f10652cbb7b503f5e00168f795d6f8be1d56"
Sep 16 18:42:31.709246 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:42:31.709217    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7036109f16009966b4cf2a119d6f10652cbb7b503f5e00168f795d6f8be1d56\": container with ID starting with d7036109f16009966b4cf2a119d6f10652cbb7b503f5e00168f795d6f8be1d56 not found: ID does not exist" containerID="d7036109f16009966b4cf2a119d6f10652cbb7b503f5e00168f795d6f8be1d56"
Sep 16 18:42:31.709361 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.709255    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7036109f16009966b4cf2a119d6f10652cbb7b503f5e00168f795d6f8be1d56"} err="failed to get container status \"d7036109f16009966b4cf2a119d6f10652cbb7b503f5e00168f795d6f8be1d56\": rpc error: code = NotFound desc = could not find container \"d7036109f16009966b4cf2a119d6f10652cbb7b503f5e00168f795d6f8be1d56\": container with ID starting with d7036109f16009966b4cf2a119d6f10652cbb7b503f5e00168f795d6f8be1d56 not found: ID does not exist"
Sep 16 18:42:31.712102 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.712080    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"]
Sep 16 18:42:31.716469 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.716442    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-jwn46"]
Sep 16 18:42:31.727834 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.727807    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc"]
Sep 16 18:42:31.734162 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:31.734139    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4pfhc"]
Sep 16 18:42:33.081989 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:33.081949    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43f2c681-99f2-4f0c-821f-8f204e875eaa" path="/var/lib/kubelet/pods/43f2c681-99f2-4f0c-821f-8f204e875eaa/volumes"
Sep 16 18:42:33.082469 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:33.082424    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ecd4d0e-8799-446d-afce-23696a05dfe9" path="/var/lib/kubelet/pods/6ecd4d0e-8799-446d-afce-23696a05dfe9/volumes"
Sep 16 18:42:33.717888 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:33.717834    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:33.819153 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:33.819113    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:39.540761 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:39.540720    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:39.633173 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:39.633133    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:40.227766 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:40.227675    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:40.359272 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:40.359235    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:59.560858 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:42:59.560821    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:26.995749 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:26.995713    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:29.217477 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:29.217441    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:30.562254 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:30.562215    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-klt7h/must-gather-psmk2"]
Sep 16 18:43:30.562657 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:30.562548    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43f2c681-99f2-4f0c-821f-8f204e875eaa" containerName="storage-initializer"
Sep 16 18:43:30.562657 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:30.562559    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="43f2c681-99f2-4f0c-821f-8f204e875eaa" containerName="storage-initializer"
Sep 16 18:43:30.562657 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:30.562572    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0a85863-b59f-44f4-9d33-de426efa8221" containerName="istio-proxy"
Sep 16 18:43:30.562657 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:30.562578    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0a85863-b59f-44f4-9d33-de426efa8221" containerName="istio-proxy"
Sep 16 18:43:30.562657 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:30.562585    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43f2c681-99f2-4f0c-821f-8f204e875eaa" containerName="main"
Sep 16 18:43:30.562657 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:30.562591    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="43f2c681-99f2-4f0c-821f-8f204e875eaa" containerName="main"
Sep 16 18:43:30.562657 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:30.562599    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ecd4d0e-8799-446d-afce-23696a05dfe9" containerName="main"
Sep 16 18:43:30.562657 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:30.562605    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ecd4d0e-8799-446d-afce-23696a05dfe9" containerName="main"
Sep 16 18:43:30.562657 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:30.562613    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0a85863-b59f-44f4-9d33-de426efa8221" containerName="istio-proxy"
Sep 16 18:43:30.562657 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:30.562619    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0a85863-b59f-44f4-9d33-de426efa8221" containerName="istio-proxy"
Sep 16 18:43:30.563011 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:30.562672    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0a85863-b59f-44f4-9d33-de426efa8221" containerName="istio-proxy"
Sep 16 18:43:30.563011 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:30.562680    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="43f2c681-99f2-4f0c-821f-8f204e875eaa" containerName="main"
Sep 16 18:43:30.563011 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:30.562688    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ecd4d0e-8799-446d-afce-23696a05dfe9" containerName="main"
Sep 16 18:43:30.563011 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:30.562781    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0a85863-b59f-44f4-9d33-de426efa8221" containerName="istio-proxy"
Sep 16 18:43:30.565770 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:30.565742    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-klt7h/must-gather-psmk2"
Sep 16 18:43:30.568669 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:30.568639    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-klt7h\"/\"kube-root-ca.crt\""
Sep 16 18:43:30.568812 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:30.568791    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-klt7h\"/\"default-dockercfg-v8rbr\""
Sep 16 18:43:30.569719 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:30.569698    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-klt7h\"/\"openshift-service-ca.crt\""
Sep 16 18:43:30.583393 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:30.583365    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-klt7h/must-gather-psmk2"]
Sep 16 18:43:30.595101 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:30.595078    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:30.633843 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:30.633801    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8-must-gather-output\") pod \"must-gather-psmk2\" (UID: \"7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8\") " pod="openshift-must-gather-klt7h/must-gather-psmk2"
Sep 16 18:43:30.634013 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:30.633880    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqtb7\" (UniqueName: \"kubernetes.io/projected/7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8-kube-api-access-bqtb7\") pod \"must-gather-psmk2\" (UID: \"7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8\") " pod="openshift-must-gather-klt7h/must-gather-psmk2"
Sep 16 18:43:30.685052 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:30.685021    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:30.734805 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:30.734770    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8-must-gather-output\") pod \"must-gather-psmk2\" (UID: \"7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8\") " pod="openshift-must-gather-klt7h/must-gather-psmk2"
Sep 16 18:43:30.734999 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:30.734823    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqtb7\" (UniqueName: \"kubernetes.io/projected/7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8-kube-api-access-bqtb7\") pod \"must-gather-psmk2\" (UID: \"7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8\") " pod="openshift-must-gather-klt7h/must-gather-psmk2"
Sep 16 18:43:30.735205 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:30.735180    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8-must-gather-output\") pod \"must-gather-psmk2\" (UID: \"7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8\") " pod="openshift-must-gather-klt7h/must-gather-psmk2"
Sep 16 18:43:30.744588 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:30.744567    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqtb7\" (UniqueName: \"kubernetes.io/projected/7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8-kube-api-access-bqtb7\") pod \"must-gather-psmk2\" (UID: \"7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8\") " pod="openshift-must-gather-klt7h/must-gather-psmk2"
Sep 16 18:43:30.880663 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:30.880566    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-klt7h/must-gather-psmk2"
Sep 16 18:43:31.163640 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:31.163608    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-klt7h/must-gather-psmk2"]
Sep 16 18:43:31.165803 ip-10-0-136-122 kubenswrapper[2577]: W0916 18:43:31.165770    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e9f94d6_f95a_4dd2_8e25_66c22a0c2df8.slice/crio-f87fd7ed17d7456eaf07d6a0c50436020e1018a93e945bbd3e6366baa3473a43 WatchSource:0}: Error finding container f87fd7ed17d7456eaf07d6a0c50436020e1018a93e945bbd3e6366baa3473a43: Status 404 returned error can't find the container with id f87fd7ed17d7456eaf07d6a0c50436020e1018a93e945bbd3e6366baa3473a43
Sep 16 18:43:31.167620 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:31.167601    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:43:31.993609 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:31.993568    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-klt7h/must-gather-psmk2" event={"ID":"7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8","Type":"ContainerStarted","Data":"f87fd7ed17d7456eaf07d6a0c50436020e1018a93e945bbd3e6366baa3473a43"}
Sep 16 18:43:33.584148 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:33.584114    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:35.574580 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:35.574543    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:38.036354 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:38.036263    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-klt7h/must-gather-psmk2" event={"ID":"7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8","Type":"ContainerStarted","Data":"8af4a7a4f13b68582dac76176d5b3ccaa3da4dfc4d120f0c6de4e3be7c877c47"}
Sep 16 18:43:38.036354 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:38.036312    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-klt7h/must-gather-psmk2" event={"ID":"7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8","Type":"ContainerStarted","Data":"e2bff1b357aafee698e43405ca00f3e4fb4411052e02f5c13508fe2b86a27fff"}
Sep 16 18:43:38.055834 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:38.055760    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-klt7h/must-gather-psmk2" podStartSLOduration=1.84747258 podStartE2EDuration="8.055738219s" totalImagesPullingTime="6.208265639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:38.053792015 +0000 UTC m=+4739.416470802" watchObservedRunningTime="2026-09-16 18:43:38.055738219 +0000 UTC m=+4739.418417005"
Sep 16 18:43:39.710142 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:39.710106    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:39.820925 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:39.820888    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:41.890933 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:41.890888    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:42.013486 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:42.013453    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:53.555629 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:53.555572    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8584f4d7d-4fmb9_448e3488-54f3-43e2-800e-de6b74a9b4f8/router/0.log"
Sep 16 18:43:54.441273 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:54.441227    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8584f4d7d-4fmb9_448e3488-54f3-43e2-800e-de6b74a9b4f8/router/0.log"
Sep 16 18:43:55.828095 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:55.828061    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-drm6x_7f849eb8-5b46-48b5-b313-f2fd773c467e/manager/0.log"
Sep 16 18:43:55.846662 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:55.846637    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-45f5c_191cce97-72b1-40a1-a139-5a48ab187aa2/manager/0.log"
Sep 16 18:43:56.052101 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:56.052070    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-vd9tf_3b3e0345-ec43-4b4e-ba85-73f7cbb3e7d1/limitador/0.log"
Sep 16 18:43:56.068788 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:43:56.068760    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-vd7dc_0a064f1b-b82e-4155-ba16-58fac8703bcd/manager/0.log"
Sep 16 18:44:01.175274 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:01.175184    2577 generic.go:356] "Generic (PLEG): container finished" podID="7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8" containerID="e2bff1b357aafee698e43405ca00f3e4fb4411052e02f5c13508fe2b86a27fff" exitCode=0
Sep 16 18:44:01.175734 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:01.175265    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-klt7h/must-gather-psmk2" event={"ID":"7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8","Type":"ContainerDied","Data":"e2bff1b357aafee698e43405ca00f3e4fb4411052e02f5c13508fe2b86a27fff"}
Sep 16 18:44:01.175734 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:01.175605    2577 scope.go:117] "RemoveContainer" containerID="e2bff1b357aafee698e43405ca00f3e4fb4411052e02f5c13508fe2b86a27fff"
Sep 16 18:44:01.294257 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:01.294221    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-klt7h_must-gather-psmk2_7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8/gather/0.log"
Sep 16 18:44:01.819574 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:01.819526    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:01.960257 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:01.960221    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wzhjv/must-gather-mnp6l"]
Sep 16 18:44:01.964663 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:01.964640    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wzhjv/must-gather-mnp6l"
Sep 16 18:44:01.967355 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:01.967323    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wzhjv\"/\"kube-root-ca.crt\""
Sep 16 18:44:01.968595 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:01.968577    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wzhjv\"/\"default-dockercfg-d5znf\""
Sep 16 18:44:01.968705 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:01.968576    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wzhjv\"/\"openshift-service-ca.crt\""
Sep 16 18:44:01.977798 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:01.977770    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wzhjv/must-gather-mnp6l"]
Sep 16 18:44:01.988578 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:01.988552    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:02.078104 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:02.078017    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:02.106141 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:02.106102    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v6dj\" (UniqueName: \"kubernetes.io/projected/08788d95-9e2e-48f7-82c8-8c6110c105bc-kube-api-access-5v6dj\") pod \"must-gather-mnp6l\" (UID: \"08788d95-9e2e-48f7-82c8-8c6110c105bc\") " pod="openshift-must-gather-wzhjv/must-gather-mnp6l"
Sep 16 18:44:02.106287 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:02.106150    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/08788d95-9e2e-48f7-82c8-8c6110c105bc-must-gather-output\") pod \"must-gather-mnp6l\" (UID: \"08788d95-9e2e-48f7-82c8-8c6110c105bc\") " pod="openshift-must-gather-wzhjv/must-gather-mnp6l"
Sep 16 18:44:02.207081 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:02.207042    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v6dj\" (UniqueName: \"kubernetes.io/projected/08788d95-9e2e-48f7-82c8-8c6110c105bc-kube-api-access-5v6dj\") pod \"must-gather-mnp6l\" (UID: \"08788d95-9e2e-48f7-82c8-8c6110c105bc\") " pod="openshift-must-gather-wzhjv/must-gather-mnp6l"
Sep 16 18:44:02.207552 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:02.207113    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/08788d95-9e2e-48f7-82c8-8c6110c105bc-must-gather-output\") pod \"must-gather-mnp6l\" (UID: \"08788d95-9e2e-48f7-82c8-8c6110c105bc\") " pod="openshift-must-gather-wzhjv/must-gather-mnp6l"
Sep 16 18:44:02.207552 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:02.207512    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/08788d95-9e2e-48f7-82c8-8c6110c105bc-must-gather-output\") pod \"must-gather-mnp6l\" (UID: \"08788d95-9e2e-48f7-82c8-8c6110c105bc\") " pod="openshift-must-gather-wzhjv/must-gather-mnp6l"
Sep 16 18:44:02.216663 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:02.216634    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v6dj\" (UniqueName: \"kubernetes.io/projected/08788d95-9e2e-48f7-82c8-8c6110c105bc-kube-api-access-5v6dj\") pod \"must-gather-mnp6l\" (UID: \"08788d95-9e2e-48f7-82c8-8c6110c105bc\") " pod="openshift-must-gather-wzhjv/must-gather-mnp6l"
Sep 16 18:44:02.273613 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:02.273573    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wzhjv/must-gather-mnp6l"
Sep 16 18:44:02.552636 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:02.552602    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wzhjv/must-gather-mnp6l"]
Sep 16 18:44:02.553091 ip-10-0-136-122 kubenswrapper[2577]: W0916 18:44:02.553063    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08788d95_9e2e_48f7_82c8_8c6110c105bc.slice/crio-84cb4996ccbf4fac3faf408a7018ebfeb8de256e8225dbd6551223009873ec1d WatchSource:0}: Error finding container 84cb4996ccbf4fac3faf408a7018ebfeb8de256e8225dbd6551223009873ec1d: Status 404 returned error can't find the container with id 84cb4996ccbf4fac3faf408a7018ebfeb8de256e8225dbd6551223009873ec1d
Sep 16 18:44:03.189616 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:03.189574    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wzhjv/must-gather-mnp6l" event={"ID":"08788d95-9e2e-48f7-82c8-8c6110c105bc","Type":"ContainerStarted","Data":"84cb4996ccbf4fac3faf408a7018ebfeb8de256e8225dbd6551223009873ec1d"}
Sep 16 18:44:04.196832 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:04.196789    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wzhjv/must-gather-mnp6l" event={"ID":"08788d95-9e2e-48f7-82c8-8c6110c105bc","Type":"ContainerStarted","Data":"f3375ee34ca981519335683341482ce3c7e60e16b26c2e1a9582c5a426814e6f"}
Sep 16 18:44:04.197258 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:04.196848    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wzhjv/must-gather-mnp6l" event={"ID":"08788d95-9e2e-48f7-82c8-8c6110c105bc","Type":"ContainerStarted","Data":"e4abf314d74ef7266bc56d31fe2310e1231ba07825b6382ff08e13a17cfa6ed4"}
Sep 16 18:44:04.217303 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:04.217230    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wzhjv/must-gather-mnp6l" podStartSLOduration=2.353291279 podStartE2EDuration="3.217208028s" totalImagesPullingTime="863.916749ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:04.214759466 +0000 UTC m=+4765.577438251" watchObservedRunningTime="2026-09-16 18:44:04.217208028 +0000 UTC m=+4765.579886814"
Sep 16 18:44:05.126355 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:05.126320    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6dj5q_451de579-98aa-48be-aeee-cf7d09759d3e/global-pull-secret-syncer/0.log"
Sep 16 18:44:05.214895 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:05.214841    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log"
Sep 16 18:44:05.360710 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:05.360657    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-122.ec2.internal_b8e15c478aefab14292c7bbfa4653ed5/haproxy/0.log"
Sep 16 18:44:06.797579 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:06.797530    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-klt7h/must-gather-psmk2"]
Sep 16 18:44:06.798112 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:06.797782    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-klt7h/must-gather-psmk2" podUID="7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8" containerName="copy" containerID="cri-o://8af4a7a4f13b68582dac76176d5b3ccaa3da4dfc4d120f0c6de4e3be7c877c47" gracePeriod=2
Sep 16 18:44:06.804114 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:06.804084    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-klt7h/must-gather-psmk2"]
Sep 16 18:44:07.177541 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:07.177515    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-klt7h_must-gather-psmk2_7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8/copy/0.log"
Sep 16 18:44:07.181026 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:07.180999    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-klt7h/must-gather-psmk2"
Sep 16 18:44:07.218444 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:07.218410    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-klt7h_must-gather-psmk2_7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8/copy/0.log"
Sep 16 18:44:07.220442 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:07.220412    2577 generic.go:356] "Generic (PLEG): container finished" podID="7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8" containerID="8af4a7a4f13b68582dac76176d5b3ccaa3da4dfc4d120f0c6de4e3be7c877c47" exitCode=143
Sep 16 18:44:07.221710 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:07.220716    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-klt7h/must-gather-psmk2"
Sep 16 18:44:07.222224 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:07.220748    2577 scope.go:117] "RemoveContainer" containerID="8af4a7a4f13b68582dac76176d5b3ccaa3da4dfc4d120f0c6de4e3be7c877c47"
Sep 16 18:44:07.239602 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:07.239576    2577 scope.go:117] "RemoveContainer" containerID="e2bff1b357aafee698e43405ca00f3e4fb4411052e02f5c13508fe2b86a27fff"
Sep 16 18:44:07.259242 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:07.259206    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqtb7\" (UniqueName: \"kubernetes.io/projected/7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8-kube-api-access-bqtb7\") pod \"7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8\" (UID: \"7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8\") "
Sep 16 18:44:07.259557 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:07.259532    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8-must-gather-output\") pod \"7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8\" (UID: \"7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8\") "
Sep 16 18:44:07.270438 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:07.270388    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8-kube-api-access-bqtb7" (OuterVolumeSpecName: "kube-api-access-bqtb7") pod "7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8" (UID: "7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8"). InnerVolumeSpecName "kube-api-access-bqtb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:44:07.283885 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:07.281193    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8" (UID: "7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:07.293024 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:07.289932    2577 scope.go:117] "RemoveContainer" containerID="8af4a7a4f13b68582dac76176d5b3ccaa3da4dfc4d120f0c6de4e3be7c877c47"
Sep 16 18:44:07.293024 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:44:07.290547    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8af4a7a4f13b68582dac76176d5b3ccaa3da4dfc4d120f0c6de4e3be7c877c47\": container with ID starting with 8af4a7a4f13b68582dac76176d5b3ccaa3da4dfc4d120f0c6de4e3be7c877c47 not found: ID does not exist" containerID="8af4a7a4f13b68582dac76176d5b3ccaa3da4dfc4d120f0c6de4e3be7c877c47"
Sep 16 18:44:07.293024 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:07.290600    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8af4a7a4f13b68582dac76176d5b3ccaa3da4dfc4d120f0c6de4e3be7c877c47"} err="failed to get container status \"8af4a7a4f13b68582dac76176d5b3ccaa3da4dfc4d120f0c6de4e3be7c877c47\": rpc error: code = NotFound desc = could not find container \"8af4a7a4f13b68582dac76176d5b3ccaa3da4dfc4d120f0c6de4e3be7c877c47\": container with ID starting with 8af4a7a4f13b68582dac76176d5b3ccaa3da4dfc4d120f0c6de4e3be7c877c47 not found: ID does not exist"
Sep 16 18:44:07.293024 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:07.290626    2577 scope.go:117] "RemoveContainer" containerID="e2bff1b357aafee698e43405ca00f3e4fb4411052e02f5c13508fe2b86a27fff"
Sep 16 18:44:07.293024 ip-10-0-136-122 kubenswrapper[2577]: E0916 18:44:07.291015    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2bff1b357aafee698e43405ca00f3e4fb4411052e02f5c13508fe2b86a27fff\": container with ID starting with e2bff1b357aafee698e43405ca00f3e4fb4411052e02f5c13508fe2b86a27fff not found: ID does not exist" containerID="e2bff1b357aafee698e43405ca00f3e4fb4411052e02f5c13508fe2b86a27fff"
Sep 16 18:44:07.293024 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:07.291054    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2bff1b357aafee698e43405ca00f3e4fb4411052e02f5c13508fe2b86a27fff"} err="failed to get container status \"e2bff1b357aafee698e43405ca00f3e4fb4411052e02f5c13508fe2b86a27fff\": rpc error: code = NotFound desc = could not find container \"e2bff1b357aafee698e43405ca00f3e4fb4411052e02f5c13508fe2b86a27fff\": container with ID starting with e2bff1b357aafee698e43405ca00f3e4fb4411052e02f5c13508fe2b86a27fff not found: ID does not exist"
Sep 16 18:44:07.360805 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:07.360758    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqtb7\" (UniqueName: \"kubernetes.io/projected/7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8-kube-api-access-bqtb7\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:44:07.360805 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:07.360795    2577 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8-must-gather-output\") on node \"ip-10-0-136-122.ec2.internal\" DevicePath \"\""
Sep 16 18:44:07.906151 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:07.906110    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:07.982542 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:07.982480    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9k478_c910d08c-80cd-486a-98b3-518d769d149f/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:09.085909 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:09.085788    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8" path="/var/lib/kubelet/pods/7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8/volumes"
Sep 16 18:44:10.167095 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:10.167051    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-drm6x_7f849eb8-5b46-48b5-b313-f2fd773c467e/manager/0.log"
Sep 16 18:44:10.201821 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:10.201776    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-45f5c_191cce97-72b1-40a1-a139-5a48ab187aa2/manager/0.log"
Sep 16 18:44:10.488076 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:10.487998    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-vd9tf_3b3e0345-ec43-4b4e-ba85-73f7cbb3e7d1/limitador/0.log"
Sep 16 18:44:10.517116 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:10.517082    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-vd7dc_0a064f1b-b82e-4155-ba16-58fac8703bcd/manager/0.log"
Sep 16 18:44:12.210632 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:12.210592    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-sx5wx_8564b3ae-746c-4d4c-bade-c8b85d78078b/monitoring-plugin/0.log"
Sep 16 18:44:12.243518 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:12.243486    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7649p_d617a6d5-32c8-4728-b616-b7872e50019e/node-exporter/0.log"
Sep 16 18:44:12.283770 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:12.283734    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7649p_d617a6d5-32c8-4728-b616-b7872e50019e/kube-rbac-proxy/0.log"
Sep 16 18:44:12.320604 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:12.320572    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7649p_d617a6d5-32c8-4728-b616-b7872e50019e/init-textfile/0.log"
Sep 16 18:44:13.019996 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:13.019816    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-776b68d986-flcr6_73d2b32c-116f-4692-bf1e-f75beae96d63/telemeter-client/0.log"
Sep 16 18:44:13.049498 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:13.049472    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-776b68d986-flcr6_73d2b32c-116f-4692-bf1e-f75beae96d63/reload/0.log"
Sep 16 18:44:13.075836 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:13.075805    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-776b68d986-flcr6_73d2b32c-116f-4692-bf1e-f75beae96d63/kube-rbac-proxy/0.log"
Sep 16 18:44:15.172942 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:15.172897    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wzhjv/perf-node-gather-daemonset-wbl98"]
Sep 16 18:44:15.173530 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:15.173447    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8" containerName="copy"
Sep 16 18:44:15.173530 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:15.173467    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8" containerName="copy"
Sep 16 18:44:15.173530 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:15.173492    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8" containerName="gather"
Sep 16 18:44:15.173530 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:15.173501    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8" containerName="gather"
Sep 16 18:44:15.173758 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:15.173577    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8" containerName="copy"
Sep 16 18:44:15.173758 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:15.173592    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e9f94d6-f95a-4dd2-8e25-66c22a0c2df8" containerName="gather"
Sep 16 18:44:15.178596 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:15.178575    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-wbl98"
Sep 16 18:44:15.189619 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:15.189585    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wzhjv/perf-node-gather-daemonset-wbl98"]
Sep 16 18:44:15.340637 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:15.340595    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d4af0135-72d7-4ffa-94f2-ace24f9184ff-proc\") pod \"perf-node-gather-daemonset-wbl98\" (UID: \"d4af0135-72d7-4ffa-94f2-ace24f9184ff\") " pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-wbl98"
Sep 16 18:44:15.340987 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:15.340958    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdvpq\" (UniqueName: \"kubernetes.io/projected/d4af0135-72d7-4ffa-94f2-ace24f9184ff-kube-api-access-xdvpq\") pod \"perf-node-gather-daemonset-wbl98\" (UID: \"d4af0135-72d7-4ffa-94f2-ace24f9184ff\") " pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-wbl98"
Sep 16 18:44:15.341163 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:15.341141    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d4af0135-72d7-4ffa-94f2-ace24f9184ff-podres\") pod \"perf-node-gather-daemonset-wbl98\" (UID: \"d4af0135-72d7-4ffa-94f2-ace24f9184ff\") " pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-wbl98"
Sep 16 18:44:15.341468 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:15.341447    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4af0135-72d7-4ffa-94f2-ace24f9184ff-lib-modules\") pod \"perf-node-gather-daemonset-wbl98\" (UID: \"d4af0135-72d7-4ffa-94f2-ace24f9184ff\") " pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-wbl98"
Sep 16 18:44:15.341674 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:15.341637    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4af0135-72d7-4ffa-94f2-ace24f9184ff-sys\") pod \"perf-node-gather-daemonset-wbl98\" (UID: \"d4af0135-72d7-4ffa-94f2-ace24f9184ff\") " pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-wbl98"
Sep 16 18:44:15.442396 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:15.442263    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4af0135-72d7-4ffa-94f2-ace24f9184ff-lib-modules\") pod \"perf-node-gather-daemonset-wbl98\" (UID: \"d4af0135-72d7-4ffa-94f2-ace24f9184ff\") " pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-wbl98"
Sep 16 18:44:15.442396 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:15.442320    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4af0135-72d7-4ffa-94f2-ace24f9184ff-sys\") pod \"perf-node-gather-daemonset-wbl98\" (UID: \"d4af0135-72d7-4ffa-94f2-ace24f9184ff\") " pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-wbl98"
Sep 16 18:44:15.442396 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:15.442390    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d4af0135-72d7-4ffa-94f2-ace24f9184ff-proc\") pod \"perf-node-gather-daemonset-wbl98\" (UID: \"d4af0135-72d7-4ffa-94f2-ace24f9184ff\") " pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-wbl98"
Sep 16 18:44:15.442805 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:15.442405    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4af0135-72d7-4ffa-94f2-ace24f9184ff-lib-modules\") pod \"perf-node-gather-daemonset-wbl98\" (UID: \"d4af0135-72d7-4ffa-94f2-ace24f9184ff\") " pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-wbl98"
Sep 16 18:44:15.442805 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:15.442422    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdvpq\" (UniqueName: \"kubernetes.io/projected/d4af0135-72d7-4ffa-94f2-ace24f9184ff-kube-api-access-xdvpq\") pod \"perf-node-gather-daemonset-wbl98\" (UID: \"d4af0135-72d7-4ffa-94f2-ace24f9184ff\") " pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-wbl98"
Sep 16 18:44:15.442805 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:15.442447    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d4af0135-72d7-4ffa-94f2-ace24f9184ff-podres\") pod \"perf-node-gather-daemonset-wbl98\" (UID: \"d4af0135-72d7-4ffa-94f2-ace24f9184ff\") " pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-wbl98"
Sep 16 18:44:15.442805 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:15.442451    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4af0135-72d7-4ffa-94f2-ace24f9184ff-sys\") pod \"perf-node-gather-daemonset-wbl98\" (UID: \"d4af0135-72d7-4ffa-94f2-ace24f9184ff\") " pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-wbl98"
Sep 16 18:44:15.442805 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:15.442477    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d4af0135-72d7-4ffa-94f2-ace24f9184ff-proc\") pod \"perf-node-gather-daemonset-wbl98\" (UID: \"d4af0135-72d7-4ffa-94f2-ace24f9184ff\") " pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-wbl98"
Sep 16 18:44:15.442805 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:15.442590    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d4af0135-72d7-4ffa-94f2-ace24f9184ff-podres\") pod \"perf-node-gather-daemonset-wbl98\" (UID: \"d4af0135-72d7-4ffa-94f2-ace24f9184ff\") " pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-wbl98"
Sep 16 18:44:15.451248 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:15.451209    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-z2f4v_bbb73d75-b302-4ef0-b3ef-01a02b6807ba/download-server/0.log"
Sep 16 18:44:15.452473 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:15.452451    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdvpq\" (UniqueName: \"kubernetes.io/projected/d4af0135-72d7-4ffa-94f2-ace24f9184ff-kube-api-access-xdvpq\") pod \"perf-node-gather-daemonset-wbl98\" (UID: \"d4af0135-72d7-4ffa-94f2-ace24f9184ff\") " pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-wbl98"
Sep 16 18:44:15.492239 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:15.492196    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-wbl98"
Sep 16 18:44:15.793807 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:15.793680    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wzhjv/perf-node-gather-daemonset-wbl98"]
Sep 16 18:44:16.294143 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:16.294102    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-wbl98" event={"ID":"d4af0135-72d7-4ffa-94f2-ace24f9184ff","Type":"ContainerStarted","Data":"4f7200e8778de953b1c8763d0cebe2257d12d63f1e070a847a891109d38aaee9"}
Sep 16 18:44:16.294631 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:16.294168    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-wbl98" event={"ID":"d4af0135-72d7-4ffa-94f2-ace24f9184ff","Type":"ContainerStarted","Data":"4ce8a01d183a01763e5e2d559d93a9cde3eacff684707696efb523763d1e2f53"}
Sep 16 18:44:16.294631 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:16.294212    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-wbl98"
Sep 16 18:44:16.318092 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:16.318022    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-wbl98" podStartSLOduration=1.318001342 podStartE2EDuration="1.318001342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:16.313890719 +0000 UTC m=+4777.676569502" watchObservedRunningTime="2026-09-16 18:44:16.318001342 +0000 UTC m=+4777.680680128"
Sep 16 18:44:17.075759 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:17.075710    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dbvzj_34b1ddc8-ea48-4109-852d-24db0edad32c/dns/0.log"
Sep 16 18:44:17.110458 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:17.110426    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dbvzj_34b1ddc8-ea48-4109-852d-24db0edad32c/kube-rbac-proxy/0.log"
Sep 16 18:44:17.404399 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:17.404331    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jfsjw_c6e2238e-ef6c-463e-8123-4663b5961709/dns-node-resolver/0.log"
Sep 16 18:44:18.070304 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:18.070267    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-75c5879656-qjbs4_02f4fa27-fb17-4f20-9500-f2d24f08a254/registry/0.log"
Sep 16 18:44:18.096174 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:18.096126    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dwmgh_ffd13091-dadb-4b9f-bd59-53f1073acedf/node-ca/0.log"
Sep 16 18:44:19.379737 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:19.379707    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8584f4d7d-4fmb9_448e3488-54f3-43e2-800e-de6b74a9b4f8/router/0.log"
Sep 16 18:44:19.921646 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:19.921599    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n48jk_1702bbd4-a888-49db-9bb3-eb6ba31fc8f9/serve-healthcheck-canary/0.log"
Sep 16 18:44:20.388040 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:20.388007    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-6g6v2_b23b1744-c013-4acf-967c-bfc9972bde31/insights-operator/0.log"
Sep 16 18:44:20.388888 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:20.388835    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-6g6v2_b23b1744-c013-4acf-967c-bfc9972bde31/insights-operator/1.log"
Sep 16 18:44:20.407563 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:20.407537    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-49fvb_ad27164e-ba4c-429e-a321-67e56945b867/kube-rbac-proxy/0.log"
Sep 16 18:44:20.429163 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:20.429132    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-49fvb_ad27164e-ba4c-429e-a321-67e56945b867/exporter/0.log"
Sep 16 18:44:20.450427 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:20.450399    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-49fvb_ad27164e-ba4c-429e-a321-67e56945b867/extractor/0.log"
Sep 16 18:44:22.313028 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:22.312996    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-wbl98"
Sep 16 18:44:23.875775 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:23.875745    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-zzmqd_04df58f0-1d3e-486f-ae6b-5ab7f27b4596/openshift-lws-operator/0.log"
Sep 16 18:44:24.603909 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:24.603850    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-cc8594966-hqrxv_85d8ff1b-c87d-4edf-8061-756fb2301955/manager/0.log"
Sep 16 18:44:24.823264 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:24.823204    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8584f4d7d-4fmb9_448e3488-54f3-43e2-800e-de6b74a9b4f8/router/0.log"
Sep 16 18:44:25.045918 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:25.045859    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-5f574bd55d-xlhdg_8a0f5af5-641b-43af-823b-e001c9ec9e88/manager/0.log"
Sep 16 18:44:25.069741 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:25.069694    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-2p94d_64f6302e-e847-45c3-ac32-db2947e8d9af/server/0.log"
Sep 16 18:44:26.142133 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:26.142095    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-hb6sq_54bdcc45-59ee-4c6e-857f-b2fa59ca861d/manager/0.log"
Sep 16 18:44:26.177755 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:26.177713    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-ppnh5_7236084f-2910-4707-954e-02331587c5e2/s3-init/0.log"
Sep 16 18:44:26.199263 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:26.199228    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-ppnh5_7236084f-2910-4707-954e-02331587c5e2/download-hf-model/0.log"
Sep 16 18:44:26.251928 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:26.251890    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-6wwcz_356b448f-a555-4ede-a47c-c4be3e8444c8/seaweedfs/0.log"
Sep 16 18:44:26.303408 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:26.303377    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8584f4d7d-4fmb9_448e3488-54f3-43e2-800e-de6b74a9b4f8/router/0.log"
Sep 16 18:44:31.948455 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:31.948416    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-jm27w_146f8eb9-491d-4f58-a764-ba70d2197637/kube-storage-version-migrator-operator/1.log"
Sep 16 18:44:31.949406 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:31.949382    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-jm27w_146f8eb9-491d-4f58-a764-ba70d2197637/kube-storage-version-migrator-operator/0.log"
Sep 16 18:44:33.118322 ip-10-0-136-122 kubenswrapper[2577]: I0916 18:44:33.118289    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7rnft_b674965e-c124-4b4c-ae07-0e17f355f0e0/kube-multus/0.log"